[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251026 14:57:29.513394 21512 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.21.2.62:43649
I20251026 14:57:29.514977 21512 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 14:57:29.516011 21512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 14:57:29.533527 21518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 14:57:29.534245 21521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 14:57:29.533670 21519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 14:57:29.534830 21512 server_base.cc:1047] running on GCE node
I20251026 14:57:30.723522 21512 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 14:57:30.723796 21512 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 14:57:30.723950 21512 hybrid_clock.cc:648] HybridClock initialized: now 1761490650723933 us; error 0 us; skew 500 ppm
I20251026 14:57:30.724768 21512 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 14:57:30.730422 21512 webserver.cc:492] Webserver started at http://127.21.2.62:36269/ using document root <none> and password file <none>
I20251026 14:57:30.731382 21512 fs_manager.cc:362] Metadata directory not provided
I20251026 14:57:30.731585 21512 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 14:57:30.731982 21512 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 14:57:30.738494 21512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761490649341556-21512-0/minicluster-data/master-0-root/instance:
uuid: "fafcec9d4a5842d88f3d10b38b75565e"
format_stamp: "Formatted at 2025-10-26 14:57:30 on dist-test-slave-9n4f"
I20251026 14:57:30.746896 21512 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20251026 14:57:30.752394 21527 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 14:57:30.753958 21512 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20251026 14:57:30.754343 21512 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761490649341556-21512-0/minicluster-data/master-0-root
uuid: "fafcec9d4a5842d88f3d10b38b75565e"
format_stamp: "Formatted at 2025-10-26 14:57:30 on dist-test-slave-9n4f"
I20251026 14:57:30.754683 21512 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761490649341556-21512-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761490649341556-21512-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761490649341556-21512-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 14:57:30.810272 21512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 14:57:30.811793 21512 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 14:57:30.812391 21512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 14:57:30.888123 21512 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.62:43649
I20251026 14:57:30.888186 21578 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.2.62:43649 every 8 connection(s)
I20251026 14:57:30.896891 21579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:30.919307 21579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e: Bootstrap starting.
I20251026 14:57:30.925452 21579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:30.927539 21579 log.cc:826] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e: Log is configured to *not* fsync() on all Append() calls
I20251026 14:57:30.932667 21579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e: No bootstrap required, opened a new log
I20251026 14:57:30.954077 21579 raft_consensus.cc:359] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafcec9d4a5842d88f3d10b38b75565e" member_type: VOTER }
I20251026 14:57:30.954643 21579 raft_consensus.cc:385] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:30.954926 21579 raft_consensus.cc:740] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fafcec9d4a5842d88f3d10b38b75565e, State: Initialized, Role: FOLLOWER
I20251026 14:57:30.955749 21579 consensus_queue.cc:260] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafcec9d4a5842d88f3d10b38b75565e" member_type: VOTER }
I20251026 14:57:30.956243 21579 raft_consensus.cc:399] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 14:57:30.956542 21579 raft_consensus.cc:493] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 14:57:30.956897 21579 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:30.961160 21579 raft_consensus.cc:515] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafcec9d4a5842d88f3d10b38b75565e" member_type: VOTER }
I20251026 14:57:30.962003 21579 leader_election.cc:304] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fafcec9d4a5842d88f3d10b38b75565e; no voters:
I20251026 14:57:30.963799 21579 leader_election.cc:290] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 14:57:30.964152 21582 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:57:30.966460 21582 raft_consensus.cc:697] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e [term 1 LEADER]: Becoming Leader. State: Replica: fafcec9d4a5842d88f3d10b38b75565e, State: Running, Role: LEADER
I20251026 14:57:30.967465 21582 consensus_queue.cc:237] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafcec9d4a5842d88f3d10b38b75565e" member_type: VOTER }
I20251026 14:57:30.968590 21579 sys_catalog.cc:565] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e [sys.catalog]: configured and running, proceeding with master startup.
I20251026 14:57:30.978543 21583 sys_catalog.cc:455] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fafcec9d4a5842d88f3d10b38b75565e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafcec9d4a5842d88f3d10b38b75565e" member_type: VOTER } }
I20251026 14:57:30.979171 21583 sys_catalog.cc:458] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e [sys.catalog]: This master's current role is: LEADER
I20251026 14:57:30.980633 21584 sys_catalog.cc:455] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fafcec9d4a5842d88f3d10b38b75565e. Latest consensus state: current_term: 1 leader_uuid: "fafcec9d4a5842d88f3d10b38b75565e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafcec9d4a5842d88f3d10b38b75565e" member_type: VOTER } }
I20251026 14:57:30.981323 21584 sys_catalog.cc:458] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e [sys.catalog]: This master's current role is: LEADER
I20251026 14:57:30.981911 21590 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 14:57:30.994390 21590 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 14:57:30.997022 21512 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 14:57:31.013473 21590 catalog_manager.cc:1357] Generated new cluster ID: 436dc475ceb044e6961bc7c3624858fa
I20251026 14:57:31.013767 21590 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 14:57:31.039016 21590 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 14:57:31.040657 21590 catalog_manager.cc:1514] Loading token signing keys...
I20251026 14:57:31.058729 21590 catalog_manager.cc:6022] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e: Generated new TSK 0
I20251026 14:57:31.059791 21590 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 14:57:31.128607 21512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 14:57:31.137722 21601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 14:57:31.139854 21602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 14:57:31.143965 21512 server_base.cc:1047] running on GCE node
W20251026 14:57:31.144083 21604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 14:57:31.145027 21512 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 14:57:31.145237 21512 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 14:57:31.145450 21512 hybrid_clock.cc:648] HybridClock initialized: now 1761490651145428 us; error 0 us; skew 500 ppm
I20251026 14:57:31.146322 21512 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 14:57:31.150945 21512 webserver.cc:492] Webserver started at http://127.21.2.1:37111/ using document root <none> and password file <none>
I20251026 14:57:31.151566 21512 fs_manager.cc:362] Metadata directory not provided
I20251026 14:57:31.151757 21512 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 14:57:31.152052 21512 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 14:57:31.153255 21512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761490649341556-21512-0/minicluster-data/ts-0-root/instance:
uuid: "21eada14be174e129814165b15454da2"
format_stamp: "Formatted at 2025-10-26 14:57:31 on dist-test-slave-9n4f"
I20251026 14:57:31.159257 21512 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20251026 14:57:31.164638 21609 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 14:57:31.165540 21512 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20251026 14:57:31.165882 21512 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761490649341556-21512-0/minicluster-data/ts-0-root
uuid: "21eada14be174e129814165b15454da2"
format_stamp: "Formatted at 2025-10-26 14:57:31 on dist-test-slave-9n4f"
I20251026 14:57:31.166182 21512 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761490649341556-21512-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761490649341556-21512-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761490649341556-21512-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 14:57:31.189643 21512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 14:57:31.191022 21512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 14:57:31.193882 21512 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 14:57:31.198567 21512 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 14:57:31.198803 21512 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 14:57:31.199100 21512 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 14:57:31.199273 21512 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 14:57:31.309046 21512 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.1:32983
I20251026 14:57:31.309096 21671 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.2.1:32983 every 8 connection(s)
I20251026 14:57:31.317394 21512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 14:57:31.327405 21676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 14:57:31.333475 21677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 14:57:31.338258 21512 server_base.cc:1047] running on GCE node
W20251026 14:57:31.339349 21680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 14:57:31.342046 21512 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 14:57:31.342456 21512 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 14:57:31.342722 21512 hybrid_clock.cc:648] HybridClock initialized: now 1761490651342705 us; error 0 us; skew 500 ppm
I20251026 14:57:31.343626 21512 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 14:57:31.345248 21672 heartbeater.cc:344] Connected to a master server at 127.21.2.62:43649
I20251026 14:57:31.345762 21672 heartbeater.cc:461] Registering TS with master...
I20251026 14:57:31.346699 21672 heartbeater.cc:507] Master 127.21.2.62:43649 requested a full tablet report, sending...
I20251026 14:57:31.347151 21512 webserver.cc:492] Webserver started at http://127.21.2.2:38911/ using document root <none> and password file <none>
I20251026 14:57:31.347895 21512 fs_manager.cc:362] Metadata directory not provided
I20251026 14:57:31.348172 21512 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 14:57:31.348517 21512 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 14:57:31.349872 21544 ts_manager.cc:194] Registered new tserver with Master: 21eada14be174e129814165b15454da2 (127.21.2.1:32983)
I20251026 14:57:31.350260 21512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761490649341556-21512-0/minicluster-data/ts-1-root/instance:
uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
format_stamp: "Formatted at 2025-10-26 14:57:31 on dist-test-slave-9n4f"
I20251026 14:57:31.352386 21544 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52794
I20251026 14:57:31.358043 21512 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20251026 14:57:31.364161 21684 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 14:57:31.365115 21512 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20251026 14:57:31.365456 21512 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761490649341556-21512-0/minicluster-data/ts-1-root
uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
format_stamp: "Formatted at 2025-10-26 14:57:31 on dist-test-slave-9n4f"
I20251026 14:57:31.365739 21512 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761490649341556-21512-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761490649341556-21512-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761490649341556-21512-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 14:57:31.394137 21512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 14:57:31.395412 21512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 14:57:31.397593 21512 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 14:57:31.400583 21512 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 14:57:31.400805 21512 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 14:57:31.401072 21512 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 14:57:31.401235 21512 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 14:57:31.499816 21512 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.2:35377
I20251026 14:57:31.500013 21746 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.2.2:35377 every 8 connection(s)
I20251026 14:57:31.506762 21512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 14:57:31.516507 21750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 14:57:31.517685 21751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 14:57:31.524652 21747 heartbeater.cc:344] Connected to a master server at 127.21.2.62:43649
I20251026 14:57:31.525137 21747 heartbeater.cc:461] Registering TS with master...
I20251026 14:57:31.526553 21747 heartbeater.cc:507] Master 127.21.2.62:43649 requested a full tablet report, sending...
W20251026 14:57:31.528199 21753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 14:57:31.529464 21544 ts_manager.cc:194] Registered new tserver with Master: d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377)
I20251026 14:57:31.529747 21512 server_base.cc:1047] running on GCE node
I20251026 14:57:31.530885 21512 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 14:57:31.531111 21512 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 14:57:31.531286 21512 hybrid_clock.cc:648] HybridClock initialized: now 1761490651531266 us; error 0 us; skew 500 ppm
I20251026 14:57:31.531488 21544 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52806
I20251026 14:57:31.531848 21512 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 14:57:31.534950 21512 webserver.cc:492] Webserver started at http://127.21.2.3:37451/ using document root <none> and password file <none>
I20251026 14:57:31.535663 21512 fs_manager.cc:362] Metadata directory not provided
I20251026 14:57:31.535912 21512 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 14:57:31.536257 21512 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 14:57:31.537779 21512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761490649341556-21512-0/minicluster-data/ts-2-root/instance:
uuid: "cdc74a031aba4e589df4c9eb8590cc57"
format_stamp: "Formatted at 2025-10-26 14:57:31 on dist-test-slave-9n4f"
I20251026 14:57:31.544596 21512 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20251026 14:57:31.549311 21758 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 14:57:31.550346 21512 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251026 14:57:31.550726 21512 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761490649341556-21512-0/minicluster-data/ts-2-root
uuid: "cdc74a031aba4e589df4c9eb8590cc57"
format_stamp: "Formatted at 2025-10-26 14:57:31 on dist-test-slave-9n4f"
I20251026 14:57:31.551101 21512 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761490649341556-21512-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761490649341556-21512-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1761490649341556-21512-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 14:57:31.570907 21512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 14:57:31.572665 21512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 14:57:31.574484 21512 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 14:57:31.577777 21512 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 14:57:31.578074 21512 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 14:57:31.578379 21512 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 14:57:31.578595 21512 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 14:57:31.674922 21512 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.3:33457
I20251026 14:57:31.674975 21820 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.2.3:33457 every 8 connection(s)
I20251026 14:57:31.696687 21821 heartbeater.cc:344] Connected to a master server at 127.21.2.62:43649
I20251026 14:57:31.697135 21821 heartbeater.cc:461] Registering TS with master...
I20251026 14:57:31.697969 21821 heartbeater.cc:507] Master 127.21.2.62:43649 requested a full tablet report, sending...
I20251026 14:57:31.700049 21544 ts_manager.cc:194] Registered new tserver with Master: cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457)
I20251026 14:57:31.700671 21512 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.018339086s
I20251026 14:57:31.701488 21544 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52820
I20251026 14:57:31.704758 21512 test_util.cc:276] Using random seed: 9563543
I20251026 14:57:31.758212 21544 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52830:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 14:57:31.763211 21544 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 14:57:31.865594 21786 tablet_service.cc:1505] Processing CreateTablet for tablet 859bffc81a9b4a248c1b284e6bc3d41b (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION VALUES < 238609294
I20251026 14:57:31.869951 21785 tablet_service.cc:1505] Processing CreateTablet for tablet d12e45731a5146be91de773fcc96acf4 (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20251026 14:57:31.873396 21784 tablet_service.cc:1505] Processing CreateTablet for tablet 2338d7ddd3a54537a9fbc8f7fc46de45 (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20251026 14:57:31.876788 21783 tablet_service.cc:1505] Processing CreateTablet for tablet fc1d64845a9543119c22b77b1256f9f3 (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20251026 14:57:31.879737 21636 tablet_service.cc:1505] Processing CreateTablet for tablet d12e45731a5146be91de773fcc96acf4 (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20251026 14:57:31.880765 21637 tablet_service.cc:1505] Processing CreateTablet for tablet 859bffc81a9b4a248c1b284e6bc3d41b (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION VALUES < 238609294
I20251026 14:57:31.882755 21634 tablet_service.cc:1505] Processing CreateTablet for tablet fc1d64845a9543119c22b77b1256f9f3 (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20251026 14:57:31.883280 21635 tablet_service.cc:1505] Processing CreateTablet for tablet 2338d7ddd3a54537a9fbc8f7fc46de45 (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20251026 14:57:31.886657 21782 tablet_service.cc:1505] Processing CreateTablet for tablet 0e31292ae87b4a04a394a31ca556654b (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20251026 14:57:31.888670 21633 tablet_service.cc:1505] Processing CreateTablet for tablet 0e31292ae87b4a04a394a31ca556654b (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20251026 14:57:31.889139 21781 tablet_service.cc:1505] Processing CreateTablet for tablet 1ecb97a63a3c422eb2079ce004a1c7ec (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20251026 14:57:31.881253 21636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d12e45731a5146be91de773fcc96acf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:31.891755 21632 tablet_service.cc:1505] Processing CreateTablet for tablet 1ecb97a63a3c422eb2079ce004a1c7ec (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20251026 14:57:31.895112 21631 tablet_service.cc:1505] Processing CreateTablet for tablet d676fec08f2a45b8b44e1e1574606e6a (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20251026 14:57:31.897874 21632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ecb97a63a3c422eb2079ce004a1c7ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:31.898061 21630 tablet_service.cc:1505] Processing CreateTablet for tablet 90496135145648aba3eb19b0190c35c6 (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20251026 14:57:31.898681 21635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2338d7ddd3a54537a9fbc8f7fc46de45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:31.867479 21786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 859bffc81a9b4a248c1b284e6bc3d41b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:31.899416 21630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90496135145648aba3eb19b0190c35c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:31.901201 21782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e31292ae87b4a04a394a31ca556654b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:31.907801 21780 tablet_service.cc:1505] Processing CreateTablet for tablet d676fec08f2a45b8b44e1e1574606e6a (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20251026 14:57:31.910063 21784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2338d7ddd3a54537a9fbc8f7fc46de45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:31.914414 21783 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc1d64845a9543119c22b77b1256f9f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:31.916754 21785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d12e45731a5146be91de773fcc96acf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:31.916620 21633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e31292ae87b4a04a394a31ca556654b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:31.917196 21629 tablet_service.cc:1505] Processing CreateTablet for tablet d4910c8c220b4256913e4ab10a75ebe1 (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20251026 14:57:31.918535 21631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d676fec08f2a45b8b44e1e1574606e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:31.919121 21780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d676fec08f2a45b8b44e1e1574606e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:31.920133 21781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ecb97a63a3c422eb2079ce004a1c7ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:31.919185 21634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc1d64845a9543119c22b77b1256f9f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:31.921132 21629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4910c8c220b4256913e4ab10a75ebe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:31.926569 21778 tablet_service.cc:1505] Processing CreateTablet for tablet d4910c8c220b4256913e4ab10a75ebe1 (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20251026 14:57:31.928045 21778 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4910c8c220b4256913e4ab10a75ebe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:31.922016 21637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 859bffc81a9b4a248c1b284e6bc3d41b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:31.931522 21779 tablet_service.cc:1505] Processing CreateTablet for tablet 90496135145648aba3eb19b0190c35c6 (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20251026 14:57:31.938256 21779 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90496135145648aba3eb19b0190c35c6. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 14:57:31.997438 21592 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251026 14:57:32.060369 21711 tablet_service.cc:1505] Processing CreateTablet for tablet d12e45731a5146be91de773fcc96acf4 (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20251026 14:57:32.065768 21711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d12e45731a5146be91de773fcc96acf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:32.069708 21839 tablet_bootstrap.cc:492] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2: Bootstrap starting.
I20251026 14:57:32.057518 21712 tablet_service.cc:1505] Processing CreateTablet for tablet 859bffc81a9b4a248c1b284e6bc3d41b (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION VALUES < 238609294
I20251026 14:57:32.066385 21709 tablet_service.cc:1505] Processing CreateTablet for tablet fc1d64845a9543119c22b77b1256f9f3 (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20251026 14:57:32.078294 21708 tablet_service.cc:1505] Processing CreateTablet for tablet 0e31292ae87b4a04a394a31ca556654b (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20251026 14:57:32.080518 21712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 859bffc81a9b4a248c1b284e6bc3d41b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:32.073581 21710 tablet_service.cc:1505] Processing CreateTablet for tablet 2338d7ddd3a54537a9fbc8f7fc46de45 (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20251026 14:57:32.080886 21707 tablet_service.cc:1505] Processing CreateTablet for tablet 1ecb97a63a3c422eb2079ce004a1c7ec (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20251026 14:57:32.090111 21706 tablet_service.cc:1505] Processing CreateTablet for tablet d676fec08f2a45b8b44e1e1574606e6a (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20251026 14:57:32.091995 21709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc1d64845a9543119c22b77b1256f9f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:32.095427 21840 tablet_bootstrap.cc:492] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57: Bootstrap starting.
I20251026 14:57:32.095985 21705 tablet_service.cc:1505] Processing CreateTablet for tablet 90496135145648aba3eb19b0190c35c6 (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20251026 14:57:32.110687 21704 tablet_service.cc:1505] Processing CreateTablet for tablet d4910c8c220b4256913e4ab10a75ebe1 (DEFAULT_TABLE table=test-workload [id=40f2635acad740a0b14108193da1fcc1]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20251026 14:57:32.132942 21710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2338d7ddd3a54537a9fbc8f7fc46de45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:32.143316 21705 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90496135145648aba3eb19b0190c35c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:32.144750 21708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e31292ae87b4a04a394a31ca556654b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:32.156692 21839 tablet_bootstrap.cc:654] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.152401 21704 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4910c8c220b4256913e4ab10a75ebe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:32.165982 21707 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ecb97a63a3c422eb2079ce004a1c7ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:32.172215 21706 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d676fec08f2a45b8b44e1e1574606e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:57:32.185943 21840 tablet_bootstrap.cc:654] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.224656 21839 tablet_bootstrap.cc:492] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2: No bootstrap required, opened a new log
I20251026 14:57:32.225205 21839 ts_tablet_manager.cc:1403] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2: Time spent bootstrapping tablet: real 0.163s user 0.018s sys 0.026s
I20251026 14:57:32.241863 21839 raft_consensus.cc:359] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.242705 21839 raft_consensus.cc:385] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.243018 21839 raft_consensus.cc:740] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21eada14be174e129814165b15454da2, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.245570 21840 tablet_bootstrap.cc:492] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57: No bootstrap required, opened a new log
I20251026 14:57:32.246096 21840 ts_tablet_manager.cc:1403] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57: Time spent bootstrapping tablet: real 0.173s user 0.015s sys 0.025s
I20251026 14:57:32.246922 21842 tablet_bootstrap.cc:492] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5: Bootstrap starting.
I20251026 14:57:32.247134 21839 consensus_queue.cc:260] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.254194 21840 raft_consensus.cc:359] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.254979 21840 raft_consensus.cc:385] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.255299 21840 raft_consensus.cc:740] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.256079 21840 consensus_queue.cc:260] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.260160 21672 heartbeater.cc:499] Master 127.21.2.62:43649 was elected leader, sending a full tablet report...
I20251026 14:57:32.261445 21839 ts_tablet_manager.cc:1434] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2: Time spent starting tablet: real 0.035s user 0.012s sys 0.003s
I20251026 14:57:32.262427 21842 tablet_bootstrap.cc:654] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.262490 21839 tablet_bootstrap.cc:492] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2: Bootstrap starting.
I20251026 14:57:32.266142 21821 heartbeater.cc:499] Master 127.21.2.62:43649 was elected leader, sending a full tablet report...
I20251026 14:57:32.268908 21840 ts_tablet_manager.cc:1434] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57: Time spent starting tablet: real 0.023s user 0.009s sys 0.000s
I20251026 14:57:32.272573 21840 tablet_bootstrap.cc:492] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57: Bootstrap starting.
I20251026 14:57:32.275090 21839 tablet_bootstrap.cc:654] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.282785 21840 tablet_bootstrap.cc:654] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.298365 21842 tablet_bootstrap.cc:492] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5: No bootstrap required, opened a new log
I20251026 14:57:32.298967 21842 ts_tablet_manager.cc:1403] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5: Time spent bootstrapping tablet: real 0.055s user 0.020s sys 0.002s
I20251026 14:57:32.301668 21839 tablet_bootstrap.cc:492] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2: No bootstrap required, opened a new log
I20251026 14:57:32.302204 21839 ts_tablet_manager.cc:1403] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2: Time spent bootstrapping tablet: real 0.040s user 0.020s sys 0.010s
I20251026 14:57:32.303339 21842 raft_consensus.cc:359] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.306428 21842 raft_consensus.cc:385] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.305104 21839 raft_consensus.cc:359] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } }
I20251026 14:57:32.306798 21842 raft_consensus.cc:740] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.307200 21839 raft_consensus.cc:385] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.306110 21840 tablet_bootstrap.cc:492] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57: No bootstrap required, opened a new log
I20251026 14:57:32.307556 21839 raft_consensus.cc:740] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21eada14be174e129814165b15454da2, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.308341 21839 consensus_queue.cc:260] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } }
I20251026 14:57:32.308085 21840 ts_tablet_manager.cc:1403] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57: Time spent bootstrapping tablet: real 0.036s user 0.011s sys 0.004s
I20251026 14:57:32.310837 21839 ts_tablet_manager.cc:1434] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20251026 14:57:32.308421 21842 consensus_queue.cc:260] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.311892 21839 tablet_bootstrap.cc:492] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2: Bootstrap starting.
I20251026 14:57:32.314566 21840 raft_consensus.cc:359] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } }
I20251026 14:57:32.315423 21840 raft_consensus.cc:385] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.315758 21840 raft_consensus.cc:740] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Initialized, Role: FOLLOWER
W20251026 14:57:32.317979 21673 tablet.cc:2378] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 14:57:32.320858 21839 tablet_bootstrap.cc:654] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.322876 21747 heartbeater.cc:499] Master 127.21.2.62:43649 was elected leader, sending a full tablet report...
I20251026 14:57:32.323252 21842 ts_tablet_manager.cc:1434] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5: Time spent starting tablet: real 0.024s user 0.009s sys 0.002s
I20251026 14:57:32.324249 21842 tablet_bootstrap.cc:492] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5: Bootstrap starting.
I20251026 14:57:32.329535 21839 tablet_bootstrap.cc:492] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2: No bootstrap required, opened a new log
I20251026 14:57:32.330076 21839 ts_tablet_manager.cc:1403] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.004s
I20251026 14:57:32.331947 21842 tablet_bootstrap.cc:654] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.333178 21839 raft_consensus.cc:359] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.334020 21839 raft_consensus.cc:385] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.334352 21839 raft_consensus.cc:740] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21eada14be174e129814165b15454da2, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.335052 21839 consensus_queue.cc:260] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.316514 21840 consensus_queue.cc:260] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } }
I20251026 14:57:32.338770 21840 ts_tablet_manager.cc:1434] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20251026 14:57:32.341044 21839 ts_tablet_manager.cc:1434] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20251026 14:57:32.340085 21840 tablet_bootstrap.cc:492] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57: Bootstrap starting.
I20251026 14:57:32.342310 21839 tablet_bootstrap.cc:492] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2: Bootstrap starting.
I20251026 14:57:32.345619 21842 tablet_bootstrap.cc:492] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5: No bootstrap required, opened a new log
I20251026 14:57:32.346252 21842 ts_tablet_manager.cc:1403] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.004s
I20251026 14:57:32.349238 21840 tablet_bootstrap.cc:654] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.349424 21839 tablet_bootstrap.cc:654] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.353726 21842 raft_consensus.cc:359] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } }
I20251026 14:57:32.354815 21842 raft_consensus.cc:385] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.355221 21842 raft_consensus.cc:740] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.356356 21842 consensus_queue.cc:260] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } }
I20251026 14:57:32.359750 21842 ts_tablet_manager.cc:1434] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5: Time spent starting tablet: real 0.013s user 0.005s sys 0.003s
I20251026 14:57:32.361073 21842 tablet_bootstrap.cc:492] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5: Bootstrap starting.
I20251026 14:57:32.369177 21840 tablet_bootstrap.cc:492] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57: No bootstrap required, opened a new log
I20251026 14:57:32.369474 21839 tablet_bootstrap.cc:492] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2: No bootstrap required, opened a new log
I20251026 14:57:32.369728 21840 ts_tablet_manager.cc:1403] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.012s
I20251026 14:57:32.370031 21839 ts_tablet_manager.cc:1403] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.007s
I20251026 14:57:32.372625 21840 raft_consensus.cc:359] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.373001 21839 raft_consensus.cc:359] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.373838 21840 raft_consensus.cc:385] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.374051 21839 raft_consensus.cc:385] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.374171 21840 raft_consensus.cc:740] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.374388 21839 raft_consensus.cc:740] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21eada14be174e129814165b15454da2, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.375080 21839 consensus_queue.cc:260] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.375064 21840 consensus_queue.cc:260] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.378031 21842 tablet_bootstrap.cc:654] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.378157 21840 ts_tablet_manager.cc:1434] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20251026 14:57:32.379153 21840 tablet_bootstrap.cc:492] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57: Bootstrap starting.
I20251026 14:57:32.383317 21849 raft_consensus.cc:493] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:57:32.383896 21849 raft_consensus.cc:515] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.385666 21840 tablet_bootstrap.cc:654] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.393631 21839 ts_tablet_manager.cc:1434] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2: Time spent starting tablet: real 0.023s user 0.004s sys 0.004s
I20251026 14:57:32.393903 21840 tablet_bootstrap.cc:492] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57: No bootstrap required, opened a new log
I20251026 14:57:32.394543 21840 ts_tablet_manager.cc:1403] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.002s
I20251026 14:57:32.395255 21839 tablet_bootstrap.cc:492] T 1ecb97a63a3c422eb2079ce004a1c7ec P 21eada14be174e129814165b15454da2: Bootstrap starting.
I20251026 14:57:32.395144 21849 leader_election.cc:290] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457)
I20251026 14:57:32.393954 21842 tablet_bootstrap.cc:492] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5: No bootstrap required, opened a new log
I20251026 14:57:32.401958 21842 ts_tablet_manager.cc:1403] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.003s
I20251026 14:57:32.404846 21840 raft_consensus.cc:359] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.404989 21842 raft_consensus.cc:359] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.405771 21842 raft_consensus.cc:385] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.406277 21839 tablet_bootstrap.cc:654] T 1ecb97a63a3c422eb2079ce004a1c7ec P 21eada14be174e129814165b15454da2: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.406198 21842 raft_consensus.cc:740] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.407454 21842 consensus_queue.cc:260] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.405709 21840 raft_consensus.cc:385] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.409212 21840 raft_consensus.cc:740] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.410060 21840 consensus_queue.cc:260] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.418416 21840 ts_tablet_manager.cc:1434] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20251026 14:57:32.419477 21840 tablet_bootstrap.cc:492] T 0e31292ae87b4a04a394a31ca556654b P cdc74a031aba4e589df4c9eb8590cc57: Bootstrap starting.
I20251026 14:57:32.410619 21842 ts_tablet_manager.cc:1434] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20251026 14:57:32.422191 21842 tablet_bootstrap.cc:492] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5: Bootstrap starting.
I20251026 14:57:32.426563 21840 tablet_bootstrap.cc:654] T 0e31292ae87b4a04a394a31ca556654b P cdc74a031aba4e589df4c9eb8590cc57: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.429320 21842 tablet_bootstrap.cc:654] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.432235 21839 tablet_bootstrap.cc:492] T 1ecb97a63a3c422eb2079ce004a1c7ec P 21eada14be174e129814165b15454da2: No bootstrap required, opened a new log
I20251026 14:57:32.432833 21839 ts_tablet_manager.cc:1403] T 1ecb97a63a3c422eb2079ce004a1c7ec P 21eada14be174e129814165b15454da2: Time spent bootstrapping tablet: real 0.038s user 0.016s sys 0.008s
I20251026 14:57:32.436030 21839 raft_consensus.cc:359] T 1ecb97a63a3c422eb2079ce004a1c7ec P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.437008 21839 raft_consensus.cc:385] T 1ecb97a63a3c422eb2079ce004a1c7ec P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.437418 21839 raft_consensus.cc:740] T 1ecb97a63a3c422eb2079ce004a1c7ec P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21eada14be174e129814165b15454da2, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.438236 21839 consensus_queue.cc:260] T 1ecb97a63a3c422eb2079ce004a1c7ec P 21eada14be174e129814165b15454da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.441633 21839 ts_tablet_manager.cc:1434] T 1ecb97a63a3c422eb2079ce004a1c7ec P 21eada14be174e129814165b15454da2: Time spent starting tablet: real 0.008s user 0.003s sys 0.005s
I20251026 14:57:32.442829 21839 tablet_bootstrap.cc:492] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2: Bootstrap starting.
I20251026 14:57:32.446676 21842 tablet_bootstrap.cc:492] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5: No bootstrap required, opened a new log
I20251026 14:57:32.447235 21842 ts_tablet_manager.cc:1403] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.008s
I20251026 14:57:32.454257 21842 raft_consensus.cc:359] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.457404 21842 raft_consensus.cc:385] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.457989 21647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc1d64845a9543119c22b77b1256f9f3" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eada14be174e129814165b15454da2" is_pre_election: true
I20251026 14:57:32.459298 21839 tablet_bootstrap.cc:654] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.457991 21842 raft_consensus.cc:740] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.462740 21840 tablet_bootstrap.cc:492] T 0e31292ae87b4a04a394a31ca556654b P cdc74a031aba4e589df4c9eb8590cc57: No bootstrap required, opened a new log
W20251026 14:57:32.461808 21686 leader_election.cc:343] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 21eada14be174e129814165b15454da2 (127.21.2.1:32983): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:57:32.463348 21840 ts_tablet_manager.cc:1403] T 0e31292ae87b4a04a394a31ca556654b P cdc74a031aba4e589df4c9eb8590cc57: Time spent bootstrapping tablet: real 0.044s user 0.012s sys 0.010s
I20251026 14:57:32.463758 21842 consensus_queue.cc:260] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.468025 21796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc1d64845a9543119c22b77b1256f9f3" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57" is_pre_election: true
I20251026 14:57:32.467696 21840 raft_consensus.cc:359] T 0e31292ae87b4a04a394a31ca556654b P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.468905 21840 raft_consensus.cc:385] T 0e31292ae87b4a04a394a31ca556654b P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.469331 21840 raft_consensus.cc:740] T 0e31292ae87b4a04a394a31ca556654b P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.469700 21842 ts_tablet_manager.cc:1434] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5: Time spent starting tablet: real 0.022s user 0.006s sys 0.004s
I20251026 14:57:32.470350 21840 consensus_queue.cc:260] T 0e31292ae87b4a04a394a31ca556654b P cdc74a031aba4e589df4c9eb8590cc57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.473251 21845 raft_consensus.cc:493] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:57:32.473793 21845 raft_consensus.cc:515] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } }
I20251026 14:57:32.477217 21845 leader_election.cc:290] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377), 21eada14be174e129814165b15454da2 (127.21.2.1:32983)
I20251026 14:57:32.477984 21845 raft_consensus.cc:493] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:57:32.478654 21845 raft_consensus.cc:515] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.470829 21842 tablet_bootstrap.cc:492] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5: Bootstrap starting.
W20251026 14:57:32.470774 21687 leader_election.cc:343] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:57:32.481645 21845 leader_election.cc:290] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377)
I20251026 14:57:32.484134 21687 leader_election.cc:304] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters: 21eada14be174e129814165b15454da2, cdc74a031aba4e589df4c9eb8590cc57
I20251026 14:57:32.486061 21840 ts_tablet_manager.cc:1434] T 0e31292ae87b4a04a394a31ca556654b P cdc74a031aba4e589df4c9eb8590cc57: Time spent starting tablet: real 0.022s user 0.008s sys 0.000s
I20251026 14:57:32.485941 21849 raft_consensus.cc:2749] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:57:32.491956 21839 tablet_bootstrap.cc:492] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2: No bootstrap required, opened a new log
I20251026 14:57:32.492503 21839 ts_tablet_manager.cc:1403] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.006s
I20251026 14:57:32.494706 21840 tablet_bootstrap.cc:492] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57: Bootstrap starting.
I20251026 14:57:32.496126 21839 raft_consensus.cc:359] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.496910 21839 raft_consensus.cc:385] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.499643 21842 tablet_bootstrap.cc:654] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.502239 21840 tablet_bootstrap.cc:654] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.497238 21839 raft_consensus.cc:740] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21eada14be174e129814165b15454da2, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.503055 21839 consensus_queue.cc:260] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.511668 21839 ts_tablet_manager.cc:1434] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20251026 14:57:32.518632 21844 raft_consensus.cc:493] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:57:32.519212 21842 tablet_bootstrap.cc:492] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5: No bootstrap required, opened a new log
I20251026 14:57:32.519101 21844 raft_consensus.cc:515] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.519888 21842 ts_tablet_manager.cc:1403] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5: Time spent bootstrapping tablet: real 0.049s user 0.016s sys 0.010s
I20251026 14:57:32.523500 21842 raft_consensus.cc:359] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.523779 21844 leader_election.cc:290] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377), cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457)
I20251026 14:57:32.524452 21842 raft_consensus.cc:385] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.524852 21842 raft_consensus.cc:740] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.525697 21842 consensus_queue.cc:260] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.533293 21842 ts_tablet_manager.cc:1434] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20251026 14:57:32.533705 21840 tablet_bootstrap.cc:492] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57: No bootstrap required, opened a new log
I20251026 14:57:32.534288 21840 ts_tablet_manager.cc:1403] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.012s
I20251026 14:57:32.534723 21842 tablet_bootstrap.cc:492] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5: Bootstrap starting.
I20251026 14:57:32.535564 21863 raft_consensus.cc:493] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:57:32.546437 21839 tablet_bootstrap.cc:492] T d676fec08f2a45b8b44e1e1574606e6a P 21eada14be174e129814165b15454da2: Bootstrap starting.
I20251026 14:57:32.546728 21840 raft_consensus.cc:359] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.556077 21840 raft_consensus.cc:385] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.556442 21840 raft_consensus.cc:740] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.556493 21647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "859bffc81a9b4a248c1b284e6bc3d41b" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eada14be174e129814165b15454da2" is_pre_election: true
I20251026 14:57:32.557240 21647 raft_consensus.cc:2468] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdc74a031aba4e589df4c9eb8590cc57 in term 0.
I20251026 14:57:32.560164 21646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d12e45731a5146be91de773fcc96acf4" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eada14be174e129814165b15454da2" is_pre_election: true
I20251026 14:57:32.557308 21840 consensus_queue.cc:260] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.561386 21646 raft_consensus.cc:2468] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdc74a031aba4e589df4c9eb8590cc57 in term 0.
I20251026 14:57:32.554296 21863 raft_consensus.cc:515] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.564342 21760 leader_election.cc:304] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, cdc74a031aba4e589df4c9eb8590cc57; no voters:
I20251026 14:57:32.565994 21845 raft_consensus.cc:2804] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:57:32.566527 21845 raft_consensus.cc:493] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:57:32.567018 21845 raft_consensus.cc:3060] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:32.566885 21760 leader_election.cc:304] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, cdc74a031aba4e589df4c9eb8590cc57; no voters:
I20251026 14:57:32.581054 21863 leader_election.cc:290] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377)
I20251026 14:57:32.581768 21863 raft_consensus.cc:2804] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:57:32.582083 21863 raft_consensus.cc:493] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:57:32.582373 21863 raft_consensus.cc:3060] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:32.586815 21840 ts_tablet_manager.cc:1434] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57: Time spent starting tablet: real 0.052s user 0.010s sys 0.000s
I20251026 14:57:32.588137 21840 tablet_bootstrap.cc:492] T fc1d64845a9543119c22b77b1256f9f3 P cdc74a031aba4e589df4c9eb8590cc57: Bootstrap starting.
I20251026 14:57:32.589685 21646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ecb97a63a3c422eb2079ce004a1c7ec" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eada14be174e129814165b15454da2" is_pre_election: true
I20251026 14:57:32.590399 21646 raft_consensus.cc:2468] T 1ecb97a63a3c422eb2079ce004a1c7ec P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdc74a031aba4e589df4c9eb8590cc57 in term 0.
I20251026 14:57:32.599126 21760 leader_election.cc:304] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, cdc74a031aba4e589df4c9eb8590cc57; no voters:
I20251026 14:57:32.601569 21796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d12e45731a5146be91de773fcc96acf4" candidate_uuid: "21eada14be174e129814165b15454da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57" is_pre_election: true
I20251026 14:57:32.601835 21722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d12e45731a5146be91de773fcc96acf4" candidate_uuid: "21eada14be174e129814165b15454da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" is_pre_election: true
I20251026 14:57:32.602507 21722 raft_consensus.cc:2468] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21eada14be174e129814165b15454da2 in term 0.
I20251026 14:57:32.602830 21871 raft_consensus.cc:2804] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:57:32.603266 21871 raft_consensus.cc:493] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:57:32.603595 21871 raft_consensus.cc:3060] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:32.604298 21611 leader_election.cc:304] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters:
I20251026 14:57:32.605250 21844 raft_consensus.cc:2804] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:57:32.586935 21845 raft_consensus.cc:515] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } }
I20251026 14:57:32.605674 21844 raft_consensus.cc:493] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:57:32.606103 21844 raft_consensus.cc:3060] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:32.609822 21839 tablet_bootstrap.cc:654] T d676fec08f2a45b8b44e1e1574606e6a P 21eada14be174e129814165b15454da2: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.611394 21646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "859bffc81a9b4a248c1b284e6bc3d41b" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eada14be174e129814165b15454da2"
I20251026 14:57:32.612097 21646 raft_consensus.cc:3060] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:32.611089 21845 leader_election.cc:290] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 election: Requested vote from peers d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377), 21eada14be174e129814165b15454da2 (127.21.2.1:32983)
I20251026 14:57:32.610704 21840 tablet_bootstrap.cc:654] T fc1d64845a9543119c22b77b1256f9f3 P cdc74a031aba4e589df4c9eb8590cc57: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.614655 21863 raft_consensus.cc:515] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.618263 21844 raft_consensus.cc:515] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.622852 21796 raft_consensus.cc:2393] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 21eada14be174e129814165b15454da2 in current term 1: Already voted for candidate cdc74a031aba4e589df4c9eb8590cc57 in this term.
I20251026 14:57:32.624607 21842 tablet_bootstrap.cc:654] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.625815 21844 leader_election.cc:290] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [CANDIDATE]: Term 1 election: Requested vote from peers d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377), cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457)
I20251026 14:57:32.628528 21647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d12e45731a5146be91de773fcc96acf4" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eada14be174e129814165b15454da2"
I20251026 14:57:32.628798 21796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d12e45731a5146be91de773fcc96acf4" candidate_uuid: "21eada14be174e129814165b15454da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
I20251026 14:57:32.629544 21647 raft_consensus.cc:2393] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cdc74a031aba4e589df4c9eb8590cc57 in current term 1: Already voted for candidate 21eada14be174e129814165b15454da2 in this term.
I20251026 14:57:32.630026 21796 raft_consensus.cc:2393] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 21eada14be174e129814165b15454da2 in current term 1: Already voted for candidate cdc74a031aba4e589df4c9eb8590cc57 in this term.
I20251026 14:57:32.627087 21722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d12e45731a5146be91de773fcc96acf4" candidate_uuid: "21eada14be174e129814165b15454da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
I20251026 14:57:32.631181 21722 raft_consensus.cc:3060] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:32.631307 21863 leader_election.cc:290] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 election: Requested vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377)
I20251026 14:57:32.632331 21646 raft_consensus.cc:2468] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc74a031aba4e589df4c9eb8590cc57 in term 1.
I20251026 14:57:32.630553 21871 raft_consensus.cc:515] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.637079 21718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "859bffc81a9b4a248c1b284e6bc3d41b" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
I20251026 14:57:32.638015 21718 raft_consensus.cc:3060] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:32.653952 21720 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d12e45731a5146be91de773fcc96acf4" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" is_pre_election: true
I20251026 14:57:32.643239 21719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ecb97a63a3c422eb2079ce004a1c7ec" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" is_pre_election: true
I20251026 14:57:32.660650 21863 raft_consensus.cc:493] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:57:32.661408 21863 raft_consensus.cc:515] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.650516 21721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "859bffc81a9b4a248c1b284e6bc3d41b" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" is_pre_election: true
I20251026 14:57:32.663453 21871 leader_election.cc:290] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 election: Requested vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377)
I20251026 14:57:32.663290 21717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d12e45731a5146be91de773fcc96acf4" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
I20251026 14:57:32.668771 21722 raft_consensus.cc:2468] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21eada14be174e129814165b15454da2 in term 1.
I20251026 14:57:32.668877 21863 leader_election.cc:290] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377)
I20251026 14:57:32.670405 21611 leader_election.cc:304] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21eada14be174e129814165b15454da2, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters: cdc74a031aba4e589df4c9eb8590cc57
I20251026 14:57:32.671218 21718 raft_consensus.cc:2468] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc74a031aba4e589df4c9eb8590cc57 in term 1.
I20251026 14:57:32.671561 21844 raft_consensus.cc:2804] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:57:32.662460 21760 leader_election.cc:304] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, cdc74a031aba4e589df4c9eb8590cc57; no voters:
I20251026 14:57:32.672564 21863 raft_consensus.cc:2804] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:57:32.672998 21863 raft_consensus.cc:697] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 1 LEADER]: Becoming Leader. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Running, Role: LEADER
I20251026 14:57:32.673835 21863 consensus_queue.cc:237] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } }
W20251026 14:57:32.678905 21760 leader_election.cc:343] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:57:32.679468 21718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ecb97a63a3c422eb2079ce004a1c7ec" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
I20251026 14:57:32.679891 21647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90496135145648aba3eb19b0190c35c6" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eada14be174e129814165b15454da2" is_pre_election: true
I20251026 14:57:32.679493 21646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ecb97a63a3c422eb2079ce004a1c7ec" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eada14be174e129814165b15454da2"
I20251026 14:57:32.680825 21760 leader_election.cc:304] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cdc74a031aba4e589df4c9eb8590cc57; no voters: 21eada14be174e129814165b15454da2, d2d4f51a45dc439ea67c2fe81ed6b1a5
I20251026 14:57:32.681099 21646 raft_consensus.cc:3060] T 1ecb97a63a3c422eb2079ce004a1c7ec P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Advancing to term 1
W20251026 14:57:32.682526 21760 leader_election.cc:343] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 21eada14be174e129814165b15454da2 (127.21.2.1:32983): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:57:32.683382 21863 raft_consensus.cc:2749] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20251026 14:57:32.684239 21760 leader_election.cc:343] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:57:32.685798 21722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90496135145648aba3eb19b0190c35c6" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" is_pre_election: true
I20251026 14:57:32.690385 21722 raft_consensus.cc:2468] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdc74a031aba4e589df4c9eb8590cc57 in term 0.
I20251026 14:57:32.692327 21646 raft_consensus.cc:2468] T 1ecb97a63a3c422eb2079ce004a1c7ec P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc74a031aba4e589df4c9eb8590cc57 in term 1.
I20251026 14:57:32.693929 21760 leader_election.cc:304] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cdc74a031aba4e589df4c9eb8590cc57, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters: 21eada14be174e129814165b15454da2
I20251026 14:57:32.694778 21863 raft_consensus.cc:2804] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:57:32.695361 21863 raft_consensus.cc:493] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:57:32.695868 21863 raft_consensus.cc:3060] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:32.696830 21544 catalog_manager.cc:5649] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 reported cstate change: term changed from 0 to 1, leader changed from <none> to cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "cdc74a031aba4e589df4c9eb8590cc57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: UNKNOWN } } }
I20251026 14:57:32.698405 21760 leader_election.cc:304] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21eada14be174e129814165b15454da2, cdc74a031aba4e589df4c9eb8590cc57; no voters: d2d4f51a45dc439ea67c2fe81ed6b1a5
I20251026 14:57:32.699466 21871 raft_consensus.cc:2804] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:57:32.699950 21871 raft_consensus.cc:697] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [term 1 LEADER]: Becoming Leader. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Running, Role: LEADER
I20251026 14:57:32.700282 21840 tablet_bootstrap.cc:492] T fc1d64845a9543119c22b77b1256f9f3 P cdc74a031aba4e589df4c9eb8590cc57: No bootstrap required, opened a new log
I20251026 14:57:32.700769 21871 consensus_queue.cc:237] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.703250 21839 tablet_bootstrap.cc:492] T d676fec08f2a45b8b44e1e1574606e6a P 21eada14be174e129814165b15454da2: No bootstrap required, opened a new log
I20251026 14:57:32.703930 21839 ts_tablet_manager.cc:1403] T d676fec08f2a45b8b44e1e1574606e6a P 21eada14be174e129814165b15454da2: Time spent bootstrapping tablet: real 0.158s user 0.013s sys 0.017s
I20251026 14:57:32.703675 21863 raft_consensus.cc:515] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.705943 21863 leader_election.cc:290] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 election: Requested vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377)
I20251026 14:57:32.707296 21646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90496135145648aba3eb19b0190c35c6" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eada14be174e129814165b15454da2"
I20251026 14:57:32.700770 21840 ts_tablet_manager.cc:1403] T fc1d64845a9543119c22b77b1256f9f3 P cdc74a031aba4e589df4c9eb8590cc57: Time spent bootstrapping tablet: real 0.113s user 0.027s sys 0.041s
W20251026 14:57:32.709546 21760 leader_election.cc:343] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 21eada14be174e129814165b15454da2 (127.21.2.1:32983): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:57:32.710605 21844 raft_consensus.cc:697] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 1 LEADER]: Becoming Leader. State: Replica: 21eada14be174e129814165b15454da2, State: Running, Role: LEADER
I20251026 14:57:32.711196 21722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90496135145648aba3eb19b0190c35c6" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
I20251026 14:57:32.712591 21722 raft_consensus.cc:3060] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:32.712615 21844 consensus_queue.cc:237] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.723626 21842 tablet_bootstrap.cc:492] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5: No bootstrap required, opened a new log
I20251026 14:57:32.724211 21842 ts_tablet_manager.cc:1403] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5: Time spent bootstrapping tablet: real 0.190s user 0.008s sys 0.022s
I20251026 14:57:32.728268 21840 raft_consensus.cc:359] T fc1d64845a9543119c22b77b1256f9f3 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.729106 21840 raft_consensus.cc:385] T fc1d64845a9543119c22b77b1256f9f3 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.730619 21842 raft_consensus.cc:359] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.731581 21842 raft_consensus.cc:385] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.731923 21842 raft_consensus.cc:740] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.729602 21840 raft_consensus.cc:740] T fc1d64845a9543119c22b77b1256f9f3 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.732857 21842 consensus_queue.cc:260] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.735942 21874 raft_consensus.cc:493] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:57:32.738766 21839 raft_consensus.cc:359] T d676fec08f2a45b8b44e1e1574606e6a P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.736092 21840 consensus_queue.cc:260] T fc1d64845a9543119c22b77b1256f9f3 P cdc74a031aba4e589df4c9eb8590cc57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.739895 21874 raft_consensus.cc:515] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.742517 21839 raft_consensus.cc:385] T d676fec08f2a45b8b44e1e1574606e6a P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.740968 21722 raft_consensus.cc:2468] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc74a031aba4e589df4c9eb8590cc57 in term 1.
I20251026 14:57:32.744212 21839 raft_consensus.cc:740] T d676fec08f2a45b8b44e1e1574606e6a P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21eada14be174e129814165b15454da2, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.746593 21760 leader_election.cc:304] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cdc74a031aba4e589df4c9eb8590cc57, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters: 21eada14be174e129814165b15454da2
I20251026 14:57:32.747690 21871 raft_consensus.cc:2804] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:57:32.749346 21840 ts_tablet_manager.cc:1434] T fc1d64845a9543119c22b77b1256f9f3 P cdc74a031aba4e589df4c9eb8590cc57: Time spent starting tablet: real 0.040s user 0.011s sys 0.000s
I20251026 14:57:32.748556 21839 consensus_queue.cc:260] T d676fec08f2a45b8b44e1e1574606e6a P 21eada14be174e129814165b15454da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.749352 21871 raft_consensus.cc:697] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 LEADER]: Becoming Leader. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Running, Role: LEADER
I20251026 14:57:32.750422 21840 tablet_bootstrap.cc:492] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57: Bootstrap starting.
I20251026 14:57:32.746661 21543 catalog_manager.cc:5649] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21eada14be174e129814165b15454da2 (127.21.2.1). New cstate: current_term: 1 leader_uuid: "21eada14be174e129814165b15454da2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: UNKNOWN } } }
I20251026 14:57:32.750574 21871 consensus_queue.cc:237] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.751655 21839 ts_tablet_manager.cc:1434] T d676fec08f2a45b8b44e1e1574606e6a P 21eada14be174e129814165b15454da2: Time spent starting tablet: real 0.047s user 0.015s sys 0.002s
I20251026 14:57:32.756114 21839 tablet_bootstrap.cc:492] T 90496135145648aba3eb19b0190c35c6 P 21eada14be174e129814165b15454da2: Bootstrap starting.
I20251026 14:57:32.756136 21842 ts_tablet_manager.cc:1434] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5: Time spent starting tablet: real 0.032s user 0.011s sys 0.000s
I20251026 14:57:32.757665 21842 tablet_bootstrap.cc:492] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5: Bootstrap starting.
I20251026 14:57:32.763576 21839 tablet_bootstrap.cc:654] T 90496135145648aba3eb19b0190c35c6 P 21eada14be174e129814165b15454da2: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.767215 21842 tablet_bootstrap.cc:654] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.769992 21722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e31292ae87b4a04a394a31ca556654b" candidate_uuid: "21eada14be174e129814165b15454da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" is_pre_election: true
I20251026 14:57:32.768143 21874 leader_election.cc:290] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377), cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457)
I20251026 14:57:32.772528 21796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e31292ae87b4a04a394a31ca556654b" candidate_uuid: "21eada14be174e129814165b15454da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57" is_pre_election: true
I20251026 14:57:32.772320 21722 raft_consensus.cc:2468] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21eada14be174e129814165b15454da2 in term 0.
I20251026 14:57:32.774737 21796 raft_consensus.cc:2468] T 0e31292ae87b4a04a394a31ca556654b P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21eada14be174e129814165b15454da2 in term 0.
I20251026 14:57:32.776517 21611 leader_election.cc:304] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters:
I20251026 14:57:32.779055 21874 raft_consensus.cc:2804] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:57:32.780118 21874 raft_consensus.cc:493] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:57:32.782121 21874 raft_consensus.cc:3060] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:32.790750 21840 tablet_bootstrap.cc:654] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.813014 21871 raft_consensus.cc:493] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:57:32.813642 21871 raft_consensus.cc:515] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.816799 21646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2338d7ddd3a54537a9fbc8f7fc46de45" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eada14be174e129814165b15454da2" is_pre_election: true
I20251026 14:57:32.817579 21646 raft_consensus.cc:2468] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdc74a031aba4e589df4c9eb8590cc57 in term 0.
I20251026 14:57:32.817634 21722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2338d7ddd3a54537a9fbc8f7fc46de45" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" is_pre_election: true
I20251026 14:57:32.815039 21543 catalog_manager.cc:5649] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 reported cstate change: term changed from 0 to 1, leader changed from <none> to cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "cdc74a031aba4e589df4c9eb8590cc57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: HEALTHY } } }
I20251026 14:57:32.818914 21760 leader_election.cc:304] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, cdc74a031aba4e589df4c9eb8590cc57; no voters:
I20251026 14:57:32.819743 21863 raft_consensus.cc:2804] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251026 14:57:32.819902 21760 leader_election.cc:343] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:57:32.816947 21871 leader_election.cc:290] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377)
I20251026 14:57:32.820192 21863 raft_consensus.cc:493] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:57:32.820612 21863 raft_consensus.cc:3060] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:32.819929 21543 catalog_manager.cc:5649] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 reported cstate change: term changed from 0 to 1, leader changed from <none> to cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "cdc74a031aba4e589df4c9eb8590cc57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: HEALTHY } } }
I20251026 14:57:32.829847 21842 tablet_bootstrap.cc:492] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5: No bootstrap required, opened a new log
I20251026 14:57:32.830446 21842 ts_tablet_manager.cc:1403] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5: Time spent bootstrapping tablet: real 0.073s user 0.012s sys 0.002s
I20251026 14:57:32.816864 21874 raft_consensus.cc:515] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.833489 21842 raft_consensus.cc:359] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.834239 21842 raft_consensus.cc:385] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.834565 21842 raft_consensus.cc:740] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.835335 21842 consensus_queue.cc:260] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.835770 21874 leader_election.cc:290] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [CANDIDATE]: Term 1 election: Requested vote from peers d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377), cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457)
I20251026 14:57:32.847937 21839 tablet_bootstrap.cc:492] T 90496135145648aba3eb19b0190c35c6 P 21eada14be174e129814165b15454da2: No bootstrap required, opened a new log
I20251026 14:57:32.836676 21722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e31292ae87b4a04a394a31ca556654b" candidate_uuid: "21eada14be174e129814165b15454da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
I20251026 14:57:32.836746 21796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e31292ae87b4a04a394a31ca556654b" candidate_uuid: "21eada14be174e129814165b15454da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
I20251026 14:57:32.851419 21722 raft_consensus.cc:3060] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:32.851708 21796 raft_consensus.cc:3060] T 0e31292ae87b4a04a394a31ca556654b P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:32.856158 21842 ts_tablet_manager.cc:1434] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5: Time spent starting tablet: real 0.025s user 0.003s sys 0.004s
I20251026 14:57:32.840430 21863 raft_consensus.cc:515] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.857326 21842 tablet_bootstrap.cc:492] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5: Bootstrap starting.
I20251026 14:57:32.858421 21863 leader_election.cc:290] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 election: Requested vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377)
I20251026 14:57:32.859948 21646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2338d7ddd3a54537a9fbc8f7fc46de45" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eada14be174e129814165b15454da2"
I20251026 14:57:32.873739 21646 raft_consensus.cc:3060] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:32.874763 21722 raft_consensus.cc:2468] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21eada14be174e129814165b15454da2 in term 1.
I20251026 14:57:32.876217 21611 leader_election.cc:304] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters:
I20251026 14:57:32.877034 21874 raft_consensus.cc:2804] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:57:32.860711 21718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2338d7ddd3a54537a9fbc8f7fc46de45" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
I20251026 14:57:32.850993 21839 ts_tablet_manager.cc:1403] T 90496135145648aba3eb19b0190c35c6 P 21eada14be174e129814165b15454da2: Time spent bootstrapping tablet: real 0.096s user 0.019s sys 0.001s
I20251026 14:57:32.877540 21874 raft_consensus.cc:697] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [term 1 LEADER]: Becoming Leader. State: Replica: 21eada14be174e129814165b15454da2, State: Running, Role: LEADER
I20251026 14:57:32.877645 21718 raft_consensus.cc:3060] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:32.878401 21874 consensus_queue.cc:237] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.880018 21839 raft_consensus.cc:359] T 90496135145648aba3eb19b0190c35c6 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.880702 21839 raft_consensus.cc:385] T 90496135145648aba3eb19b0190c35c6 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.880985 21839 raft_consensus.cc:740] T 90496135145648aba3eb19b0190c35c6 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21eada14be174e129814165b15454da2, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.881644 21839 consensus_queue.cc:260] T 90496135145648aba3eb19b0190c35c6 P 21eada14be174e129814165b15454da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.865973 21796 raft_consensus.cc:2468] T 0e31292ae87b4a04a394a31ca556654b P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21eada14be174e129814165b15454da2 in term 1.
I20251026 14:57:32.869151 21840 tablet_bootstrap.cc:492] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57: No bootstrap required, opened a new log
I20251026 14:57:32.886143 21840 ts_tablet_manager.cc:1403] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57: Time spent bootstrapping tablet: real 0.136s user 0.042s sys 0.033s
I20251026 14:57:32.885957 21718 raft_consensus.cc:2468] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc74a031aba4e589df4c9eb8590cc57 in term 1.
I20251026 14:57:32.887532 21760 leader_election.cc:304] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdc74a031aba4e589df4c9eb8590cc57, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters:
I20251026 14:57:32.887845 21842 tablet_bootstrap.cc:654] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.888756 21840 raft_consensus.cc:359] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.889461 21840 raft_consensus.cc:385] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.889720 21840 raft_consensus.cc:740] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.890489 21839 ts_tablet_manager.cc:1434] T 90496135145648aba3eb19b0190c35c6 P 21eada14be174e129814165b15454da2: Time spent starting tablet: real 0.013s user 0.002s sys 0.004s
I20251026 14:57:32.890353 21840 consensus_queue.cc:260] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.891623 21839 tablet_bootstrap.cc:492] T d4910c8c220b4256913e4ab10a75ebe1 P 21eada14be174e129814165b15454da2: Bootstrap starting.
I20251026 14:57:32.893770 21871 raft_consensus.cc:2804] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:57:32.894320 21871 raft_consensus.cc:697] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 LEADER]: Becoming Leader. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Running, Role: LEADER
I20251026 14:57:32.895182 21871 consensus_queue.cc:237] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.896976 21844 raft_consensus.cc:493] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:57:32.897631 21844 raft_consensus.cc:515] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.900684 21844 leader_election.cc:290] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377), cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457)
I20251026 14:57:32.901199 21718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc1d64845a9543119c22b77b1256f9f3" candidate_uuid: "21eada14be174e129814165b15454da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" is_pre_election: true
I20251026 14:57:32.901947 21718 raft_consensus.cc:2468] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21eada14be174e129814165b15454da2 in term 0.
I20251026 14:57:32.901929 21796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc1d64845a9543119c22b77b1256f9f3" candidate_uuid: "21eada14be174e129814165b15454da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57" is_pre_election: true
I20251026 14:57:32.903378 21796 raft_consensus.cc:2468] T fc1d64845a9543119c22b77b1256f9f3 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21eada14be174e129814165b15454da2 in term 0.
I20251026 14:57:32.903294 21611 leader_election.cc:304] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters:
I20251026 14:57:32.905262 21844 raft_consensus.cc:2804] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:57:32.905802 21844 raft_consensus.cc:493] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:57:32.906234 21844 raft_consensus.cc:3060] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:32.909711 21840 ts_tablet_manager.cc:1434] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20251026 14:57:32.910733 21840 tablet_bootstrap.cc:492] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57: Bootstrap starting.
I20251026 14:57:32.913583 21646 raft_consensus.cc:2468] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc74a031aba4e589df4c9eb8590cc57 in term 1.
I20251026 14:57:32.918969 21844 raft_consensus.cc:515] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.930691 21718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc1d64845a9543119c22b77b1256f9f3" candidate_uuid: "21eada14be174e129814165b15454da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
I20251026 14:57:32.931542 21718 raft_consensus.cc:3060] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:32.931586 21796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc1d64845a9543119c22b77b1256f9f3" candidate_uuid: "21eada14be174e129814165b15454da2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
I20251026 14:57:32.932330 21796 raft_consensus.cc:3060] T fc1d64845a9543119c22b77b1256f9f3 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:32.933209 21839 tablet_bootstrap.cc:654] T d4910c8c220b4256913e4ab10a75ebe1 P 21eada14be174e129814165b15454da2: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.931318 21844 leader_election.cc:290] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [CANDIDATE]: Term 1 election: Requested vote from peers d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377), cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457)
I20251026 14:57:32.942893 21842 tablet_bootstrap.cc:492] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5: No bootstrap required, opened a new log
I20251026 14:57:32.943437 21842 ts_tablet_manager.cc:1403] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5: Time spent bootstrapping tablet: real 0.086s user 0.026s sys 0.018s
I20251026 14:57:32.946794 21840 tablet_bootstrap.cc:654] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.945477 21543 catalog_manager.cc:5649] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21eada14be174e129814165b15454da2 (127.21.2.1). New cstate: current_term: 1 leader_uuid: "21eada14be174e129814165b15454da2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: UNKNOWN } } }
I20251026 14:57:32.953050 21842 raft_consensus.cc:359] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.953799 21842 raft_consensus.cc:385] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.954102 21842 raft_consensus.cc:740] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.954741 21842 consensus_queue.cc:260] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.957646 21842 ts_tablet_manager.cc:1434] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20251026 14:57:32.958396 21842 tablet_bootstrap.cc:492] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5: Bootstrap starting.
I20251026 14:57:32.962051 21796 raft_consensus.cc:2468] T fc1d64845a9543119c22b77b1256f9f3 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21eada14be174e129814165b15454da2 in term 1.
I20251026 14:57:32.963951 21612 leader_election.cc:304] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, cdc74a031aba4e589df4c9eb8590cc57; no voters:
I20251026 14:57:32.965370 21844 raft_consensus.cc:2804] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:57:32.966063 21844 raft_consensus.cc:697] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 1 LEADER]: Becoming Leader. State: Replica: 21eada14be174e129814165b15454da2, State: Running, Role: LEADER
I20251026 14:57:32.967204 21844 consensus_queue.cc:237] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.972100 21718 raft_consensus.cc:2468] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21eada14be174e129814165b15454da2 in term 1.
I20251026 14:57:32.948329 21542 catalog_manager.cc:5649] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 reported cstate change: term changed from 0 to 1, leader changed from <none> to cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "cdc74a031aba4e589df4c9eb8590cc57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: HEALTHY } } }
I20251026 14:57:32.978574 21842 tablet_bootstrap.cc:654] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5: Neither blocks nor log segments found. Creating new log.
I20251026 14:57:32.982440 21840 tablet_bootstrap.cc:492] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57: No bootstrap required, opened a new log
I20251026 14:57:32.982946 21840 ts_tablet_manager.cc:1403] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57: Time spent bootstrapping tablet: real 0.072s user 0.007s sys 0.007s
I20251026 14:57:32.988161 21840 raft_consensus.cc:359] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.988889 21840 raft_consensus.cc:385] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.989171 21840 raft_consensus.cc:740] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.989862 21840 consensus_queue.cc:260] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.992456 21839 tablet_bootstrap.cc:492] T d4910c8c220b4256913e4ab10a75ebe1 P 21eada14be174e129814165b15454da2: No bootstrap required, opened a new log
I20251026 14:57:32.992918 21839 ts_tablet_manager.cc:1403] T d4910c8c220b4256913e4ab10a75ebe1 P 21eada14be174e129814165b15454da2: Time spent bootstrapping tablet: real 0.102s user 0.034s sys 0.009s
I20251026 14:57:32.994774 21840 ts_tablet_manager.cc:1434] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20251026 14:57:32.995556 21839 raft_consensus.cc:359] T d4910c8c220b4256913e4ab10a75ebe1 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:32.995047 21543 catalog_manager.cc:5649] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21eada14be174e129814165b15454da2 (127.21.2.1). New cstate: current_term: 1 leader_uuid: "21eada14be174e129814165b15454da2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: UNKNOWN } } }
I20251026 14:57:32.996122 21839 raft_consensus.cc:385] T d4910c8c220b4256913e4ab10a75ebe1 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:32.996382 21839 raft_consensus.cc:740] T d4910c8c220b4256913e4ab10a75ebe1 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21eada14be174e129814165b15454da2, State: Initialized, Role: FOLLOWER
I20251026 14:57:32.996954 21839 consensus_queue.cc:260] T d4910c8c220b4256913e4ab10a75ebe1 P 21eada14be174e129814165b15454da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.001084 21839 ts_tablet_manager.cc:1434] T d4910c8c220b4256913e4ab10a75ebe1 P 21eada14be174e129814165b15454da2: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20251026 14:57:33.004655 21842 tablet_bootstrap.cc:492] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5: No bootstrap required, opened a new log
I20251026 14:57:33.005149 21842 ts_tablet_manager.cc:1403] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5: Time spent bootstrapping tablet: real 0.047s user 0.014s sys 0.000s
I20251026 14:57:33.007884 21842 raft_consensus.cc:359] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.008607 21842 raft_consensus.cc:385] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:57:33.008872 21842 raft_consensus.cc:740] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Initialized, Role: FOLLOWER
I20251026 14:57:33.009544 21842 consensus_queue.cc:260] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.011639 21842 ts_tablet_manager.cc:1434] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20251026 14:57:33.013808 21592 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251026 14:57:33.053220 21849 raft_consensus.cc:493] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:57:33.053844 21849 raft_consensus.cc:515] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.055759 21849 leader_election.cc:290] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457)
I20251026 14:57:33.056559 21646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4910c8c220b4256913e4ab10a75ebe1" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eada14be174e129814165b15454da2" is_pre_election: true
I20251026 14:57:33.057236 21646 raft_consensus.cc:2468] T d4910c8c220b4256913e4ab10a75ebe1 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 0.
I20251026 14:57:33.057258 21796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4910c8c220b4256913e4ab10a75ebe1" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57" is_pre_election: true
I20251026 14:57:33.058027 21796 raft_consensus.cc:2468] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 0.
I20251026 14:57:33.058276 21686 leader_election.cc:304] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters:
I20251026 14:57:33.059098 21849 raft_consensus.cc:2804] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:57:33.059392 21849 raft_consensus.cc:493] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:57:33.059650 21849 raft_consensus.cc:3060] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:33.064303 21849 raft_consensus.cc:515] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.065992 21849 leader_election.cc:290] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 1 election: Requested vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457)
I20251026 14:57:33.066972 21646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4910c8c220b4256913e4ab10a75ebe1" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eada14be174e129814165b15454da2"
I20251026 14:57:33.067154 21796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4910c8c220b4256913e4ab10a75ebe1" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
I20251026 14:57:33.067577 21646 raft_consensus.cc:3060] T d4910c8c220b4256913e4ab10a75ebe1 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:33.067793 21796 raft_consensus.cc:3060] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:33.072547 21646 raft_consensus.cc:2468] T d4910c8c220b4256913e4ab10a75ebe1 P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 1.
I20251026 14:57:33.073598 21686 leader_election.cc:304] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters:
I20251026 14:57:33.074262 21849 raft_consensus.cc:2804] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:57:33.074661 21796 raft_consensus.cc:2468] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 1.
I20251026 14:57:33.075877 21849 raft_consensus.cc:697] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 LEADER]: Becoming Leader. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Running, Role: LEADER
I20251026 14:57:33.077179 21849 consensus_queue.cc:237] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.084347 21718 raft_consensus.cc:3060] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:33.086122 21544 catalog_manager.cc:5649] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2). New cstate: current_term: 1 leader_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: UNKNOWN } } }
I20251026 14:57:33.093220 21871 consensus_queue.cc:1048] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:57:33.107165 21845 consensus_queue.cc:1048] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:57:33.121625 21849 raft_consensus.cc:493] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:57:33.122186 21849 raft_consensus.cc:515] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.124163 21849 leader_election.cc:290] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457)
I20251026 14:57:33.125396 21647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d676fec08f2a45b8b44e1e1574606e6a" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eada14be174e129814165b15454da2" is_pre_election: true
I20251026 14:57:33.126021 21647 raft_consensus.cc:2468] T d676fec08f2a45b8b44e1e1574606e6a P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 0.
I20251026 14:57:33.125391 21796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d676fec08f2a45b8b44e1e1574606e6a" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57" is_pre_election: true
I20251026 14:57:33.127179 21796 raft_consensus.cc:2468] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 0.
I20251026 14:57:33.128228 21687 leader_election.cc:304] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdc74a031aba4e589df4c9eb8590cc57, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters:
I20251026 14:57:33.128986 21849 raft_consensus.cc:2804] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:57:33.129384 21849 raft_consensus.cc:493] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:57:33.129732 21849 raft_consensus.cc:3060] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:33.130540 21647 raft_consensus.cc:3060] T 90496135145648aba3eb19b0190c35c6 P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:33.136823 21849 raft_consensus.cc:515] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.136941 21845 consensus_queue.cc:1048] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:57:33.141453 21849 leader_election.cc:290] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 1 election: Requested vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457)
I20251026 14:57:33.142776 21796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d676fec08f2a45b8b44e1e1574606e6a" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
I20251026 14:57:33.143556 21796 raft_consensus.cc:3060] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:33.143483 21645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d676fec08f2a45b8b44e1e1574606e6a" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eada14be174e129814165b15454da2"
I20251026 14:57:33.144357 21645 raft_consensus.cc:3060] T d676fec08f2a45b8b44e1e1574606e6a P 21eada14be174e129814165b15454da2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:57:33.151506 21796 raft_consensus.cc:2468] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 1.
I20251026 14:57:33.153628 21687 leader_election.cc:304] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdc74a031aba4e589df4c9eb8590cc57, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters:
I20251026 14:57:33.154551 21849 raft_consensus.cc:2804] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:57:33.155050 21849 raft_consensus.cc:697] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 LEADER]: Becoming Leader. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Running, Role: LEADER
I20251026 14:57:33.155908 21849 consensus_queue.cc:237] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.163403 21645 raft_consensus.cc:2468] T d676fec08f2a45b8b44e1e1574606e6a P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 1.
I20251026 14:57:33.166600 21863 consensus_queue.cc:1048] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:57:33.180084 21542 catalog_manager.cc:5649] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2). New cstate: current_term: 1 leader_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: UNKNOWN } } }
I20251026 14:57:33.183775 21844 consensus_queue.cc:1048] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:57:33.185019 21871 consensus_queue.cc:1048] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:57:33.209519 21863 consensus_queue.cc:1048] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:57:33.215436 21874 consensus_queue.cc:1048] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:57:33.268317 21874 consensus_queue.cc:1048] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:57:33.279057 21844 consensus_queue.cc:1048] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:57:33.323402 21863 consensus_queue.cc:1048] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:57:33.342093 21871 consensus_queue.cc:1048] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:57:33.360360 21796 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "859bffc81a9b4a248c1b284e6bc3d41b"
dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
mode: GRACEFUL
new_leader_uuid: "21eada14be174e129814165b15454da2"
from {username='slave'} at 127.0.0.1:48116
I20251026 14:57:33.361016 21796 raft_consensus.cc:606] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 1 LEADER]: Received request to transfer leadership to TS 21eada14be174e129814165b15454da2
I20251026 14:57:33.371050 21796 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2338d7ddd3a54537a9fbc8f7fc46de45"
dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
mode: GRACEFUL
new_leader_uuid: "21eada14be174e129814165b15454da2"
from {username='slave'} at 127.0.0.1:48116
I20251026 14:57:33.371588 21796 raft_consensus.cc:606] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 LEADER]: Received request to transfer leadership to TS 21eada14be174e129814165b15454da2
I20251026 14:57:33.388528 21645 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0e31292ae87b4a04a394a31ca556654b"
dest_uuid: "21eada14be174e129814165b15454da2"
mode: GRACEFUL
new_leader_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
from {username='slave'} at 127.0.0.1:44010
I20251026 14:57:33.389230 21645 raft_consensus.cc:606] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [term 1 LEADER]: Received request to transfer leadership to TS d2d4f51a45dc439ea67c2fe81ed6b1a5
I20251026 14:57:33.394775 21796 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "1ecb97a63a3c422eb2079ce004a1c7ec"
dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
mode: GRACEFUL
new_leader_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
from {username='slave'} at 127.0.0.1:48116
I20251026 14:57:33.395295 21796 raft_consensus.cc:606] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [term 1 LEADER]: Received request to transfer leadership to TS d2d4f51a45dc439ea67c2fe81ed6b1a5
I20251026 14:57:33.402587 21796 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "90496135145648aba3eb19b0190c35c6"
dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
mode: GRACEFUL
new_leader_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
from {username='slave'} at 127.0.0.1:48116
I20251026 14:57:33.403095 21796 raft_consensus.cc:606] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 LEADER]: Received request to transfer leadership to TS d2d4f51a45dc439ea67c2fe81ed6b1a5
I20251026 14:57:33.416769 21718 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d4910c8c220b4256913e4ab10a75ebe1"
dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
mode: GRACEFUL
new_leader_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
from {username='slave'} at 127.0.0.1:35972
I20251026 14:57:33.416787 21844 consensus_queue.cc:1048] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:57:33.417539 21718 raft_consensus.cc:606] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 LEADER]: Received request to transfer leadership to TS cdc74a031aba4e589df4c9eb8590cc57
I20251026 14:57:33.429760 21874 consensus_queue.cc:1048] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:57:33.519711 21849 consensus_queue.cc:1048] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:57:33.530140 21849 consensus_queue.cc:1048] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:57:33.543577 21893 raft_consensus.cc:993] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5: : Instructing follower cdc74a031aba4e589df4c9eb8590cc57 to start an election
I20251026 14:57:33.544099 21919 raft_consensus.cc:1081] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 LEADER]: Signalling peer cdc74a031aba4e589df4c9eb8590cc57 to start an election
I20251026 14:57:33.546113 21796 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "d4910c8c220b4256913e4ab10a75ebe1"
dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
from {username='slave'} at 127.0.0.1:48104
I20251026 14:57:33.546898 21796 raft_consensus.cc:493] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:57:33.547384 21796 raft_consensus.cc:3060] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:33.553210 21796 raft_consensus.cc:515] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.554955 21796 leader_election.cc:290] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 2 election: Requested vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377)
I20251026 14:57:33.556145 21718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4910c8c220b4256913e4ab10a75ebe1" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
I20251026 14:57:33.556737 21718 raft_consensus.cc:3055] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:57:33.556974 21718 raft_consensus.cc:740] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Running, Role: LEADER
I20251026 14:57:33.557610 21718 consensus_queue.cc:260] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.558346 21645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d4910c8c220b4256913e4ab10a75ebe1" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "21eada14be174e129814165b15454da2"
I20251026 14:57:33.558898 21645 raft_consensus.cc:3060] T d4910c8c220b4256913e4ab10a75ebe1 P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:33.558856 21718 raft_consensus.cc:3060] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:33.565861 21645 raft_consensus.cc:2468] T d4910c8c220b4256913e4ab10a75ebe1 P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc74a031aba4e589df4c9eb8590cc57 in term 2.
I20251026 14:57:33.565861 21718 raft_consensus.cc:2468] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc74a031aba4e589df4c9eb8590cc57 in term 2.
I20251026 14:57:33.567413 21760 leader_election.cc:304] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, cdc74a031aba4e589df4c9eb8590cc57; no voters:
I20251026 14:57:33.568300 21845 raft_consensus.cc:2804] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:57:33.568694 21845 raft_consensus.cc:697] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57 [term 2 LEADER]: Becoming Leader. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Running, Role: LEADER
I20251026 14:57:33.569433 21845 consensus_queue.cc:237] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.575985 21542 catalog_manager.cc:5649] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57 reported cstate change: term changed from 1 to 2, leader changed from d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2) to cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3). New cstate: current_term: 2 leader_uuid: "cdc74a031aba4e589df4c9eb8590cc57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: HEALTHY } } }
I20251026 14:57:33.587873 21871 raft_consensus.cc:993] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57: : Instructing follower d2d4f51a45dc439ea67c2fe81ed6b1a5 to start an election
I20251026 14:57:33.588402 21845 raft_consensus.cc:1081] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 LEADER]: Signalling peer d2d4f51a45dc439ea67c2fe81ed6b1a5 to start an election
I20251026 14:57:33.589943 21718 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "90496135145648aba3eb19b0190c35c6"
dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
from {username='slave'} at 127.0.0.1:35952
I20251026 14:57:33.590462 21718 raft_consensus.cc:493] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:57:33.590862 21718 raft_consensus.cc:3060] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:33.594821 21871 raft_consensus.cc:993] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57: : Instructing follower 21eada14be174e129814165b15454da2 to start an election
I20251026 14:57:33.595336 21845 raft_consensus.cc:1081] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 1 LEADER]: Signalling peer 21eada14be174e129814165b15454da2 to start an election
I20251026 14:57:33.597260 21645 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "859bffc81a9b4a248c1b284e6bc3d41b"
dest_uuid: "21eada14be174e129814165b15454da2"
from {username='slave'} at 127.0.0.1:44004
I20251026 14:57:33.598037 21645 raft_consensus.cc:493] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:57:33.598052 21718 raft_consensus.cc:515] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.598680 21645 raft_consensus.cc:3060] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:33.601347 21646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90496135145648aba3eb19b0190c35c6" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "21eada14be174e129814165b15454da2"
I20251026 14:57:33.601526 21796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90496135145648aba3eb19b0190c35c6" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
I20251026 14:57:33.601933 21646 raft_consensus.cc:3060] T 90496135145648aba3eb19b0190c35c6 P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:33.602165 21796 raft_consensus.cc:3055] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:57:33.602535 21796 raft_consensus.cc:740] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Running, Role: LEADER
I20251026 14:57:33.603395 21796 consensus_queue.cc:260] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.604624 21796 raft_consensus.cc:3060] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:33.606029 21645 raft_consensus.cc:515] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } }
I20251026 14:57:33.609510 21645 leader_election.cc:290] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [CANDIDATE]: Term 2 election: Requested vote from peers cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457), d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377)
I20251026 14:57:33.609398 21795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "859bffc81a9b4a248c1b284e6bc3d41b" candidate_uuid: "21eada14be174e129814165b15454da2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
I20251026 14:57:33.610183 21795 raft_consensus.cc:3055] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:57:33.610549 21795 raft_consensus.cc:740] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Running, Role: LEADER
I20251026 14:57:33.610771 21722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "859bffc81a9b4a248c1b284e6bc3d41b" candidate_uuid: "21eada14be174e129814165b15454da2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
I20251026 14:57:33.611408 21722 raft_consensus.cc:3060] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:33.611455 21795 consensus_queue.cc:260] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } }
I20251026 14:57:33.612902 21795 raft_consensus.cc:3060] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:33.610466 21646 raft_consensus.cc:2468] T 90496135145648aba3eb19b0190c35c6 P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 2.
I20251026 14:57:33.616943 21722 raft_consensus.cc:2468] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21eada14be174e129814165b15454da2 in term 2.
I20251026 14:57:33.618492 21611 leader_election.cc:304] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters:
I20251026 14:57:33.619143 21718 leader_election.cc:290] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 2 election: Requested vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457)
I20251026 14:57:33.619508 21874 raft_consensus.cc:2804] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:57:33.620018 21874 raft_consensus.cc:697] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 2 LEADER]: Becoming Leader. State: Replica: 21eada14be174e129814165b15454da2, State: Running, Role: LEADER
I20251026 14:57:33.620127 21795 raft_consensus.cc:2468] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21eada14be174e129814165b15454da2 in term 2.
I20251026 14:57:33.620998 21874 consensus_queue.cc:237] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } }
I20251026 14:57:33.621909 21686 leader_election.cc:304] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters:
I20251026 14:57:33.622722 21919 raft_consensus.cc:2804] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:57:33.623207 21919 raft_consensus.cc:697] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 LEADER]: Becoming Leader. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Running, Role: LEADER
I20251026 14:57:33.622960 21796 raft_consensus.cc:2468] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 2.
I20251026 14:57:33.624056 21919 consensus_queue.cc:237] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.630256 21542 catalog_manager.cc:5649] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 reported cstate change: term changed from 1 to 2, leader changed from cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3) to 21eada14be174e129814165b15454da2 (127.21.2.1). New cstate: current_term: 2 leader_uuid: "21eada14be174e129814165b15454da2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: HEALTHY } } }
I20251026 14:57:33.633209 21543 catalog_manager.cc:5649] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 reported cstate change: term changed from 1 to 2, leader changed from cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3) to d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2). New cstate: current_term: 2 leader_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: UNKNOWN } } }
I20251026 14:57:33.710361 21871 raft_consensus.cc:993] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57: : Instructing follower d2d4f51a45dc439ea67c2fe81ed6b1a5 to start an election
I20251026 14:57:33.710906 21845 raft_consensus.cc:1081] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [term 1 LEADER]: Signalling peer d2d4f51a45dc439ea67c2fe81ed6b1a5 to start an election
I20251026 14:57:33.712477 21722 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "1ecb97a63a3c422eb2079ce004a1c7ec"
dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
from {username='slave'} at 127.0.0.1:35952
I20251026 14:57:33.712996 21722 raft_consensus.cc:493] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:57:33.713318 21722 raft_consensus.cc:3060] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:33.717715 21919 consensus_queue.cc:1048] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:57:33.720077 21722 raft_consensus.cc:515] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.722396 21722 leader_election.cc:290] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 2 election: Requested vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457)
I20251026 14:57:33.723067 21646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ecb97a63a3c422eb2079ce004a1c7ec" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "21eada14be174e129814165b15454da2"
I20251026 14:57:33.723672 21646 raft_consensus.cc:3060] T 1ecb97a63a3c422eb2079ce004a1c7ec P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:33.728667 21795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1ecb97a63a3c422eb2079ce004a1c7ec" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
I20251026 14:57:33.729408 21795 raft_consensus.cc:3055] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:57:33.732163 21795 raft_consensus.cc:740] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Running, Role: LEADER
I20251026 14:57:33.733197 21795 consensus_queue.cc:260] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.734663 21795 raft_consensus.cc:3060] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:33.734843 21893 consensus_queue.cc:1048] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:57:33.744177 21646 raft_consensus.cc:2468] T 1ecb97a63a3c422eb2079ce004a1c7ec P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 2.
I20251026 14:57:33.745225 21795 raft_consensus.cc:2468] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 2.
I20251026 14:57:33.745544 21686 leader_election.cc:304] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters:
I20251026 14:57:33.746436 21849 raft_consensus.cc:2804] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:57:33.746908 21849 raft_consensus.cc:697] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 LEADER]: Becoming Leader. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Running, Role: LEADER
I20251026 14:57:33.747740 21849 consensus_queue.cc:237] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.756498 21543 catalog_manager.cc:5649] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5 reported cstate change: term changed from 1 to 2, leader changed from cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3) to d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2). New cstate: current_term: 2 leader_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: UNKNOWN } } }
I20251026 14:57:33.761539 21845 raft_consensus.cc:993] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57: : Instructing follower 21eada14be174e129814165b15454da2 to start an election
I20251026 14:57:33.762035 21845 raft_consensus.cc:1081] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 LEADER]: Signalling peer 21eada14be174e129814165b15454da2 to start an election
I20251026 14:57:33.763749 21646 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2338d7ddd3a54537a9fbc8f7fc46de45"
dest_uuid: "21eada14be174e129814165b15454da2"
from {username='slave'} at 127.0.0.1:44004
I20251026 14:57:33.764392 21646 raft_consensus.cc:493] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:57:33.764743 21646 raft_consensus.cc:3060] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:33.772042 21646 raft_consensus.cc:515] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.774698 21646 leader_election.cc:290] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [CANDIDATE]: Term 2 election: Requested vote from peers d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377), cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457)
I20251026 14:57:33.774942 21722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2338d7ddd3a54537a9fbc8f7fc46de45" candidate_uuid: "21eada14be174e129814165b15454da2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
I20251026 14:57:33.775547 21722 raft_consensus.cc:3060] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:33.775074 21795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2338d7ddd3a54537a9fbc8f7fc46de45" candidate_uuid: "21eada14be174e129814165b15454da2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
I20251026 14:57:33.776310 21795 raft_consensus.cc:3055] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:57:33.776630 21795 raft_consensus.cc:740] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Running, Role: LEADER
I20251026 14:57:33.777488 21795 consensus_queue.cc:260] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.778702 21795 raft_consensus.cc:3060] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:33.780500 21722 raft_consensus.cc:2468] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21eada14be174e129814165b15454da2 in term 2.
I20251026 14:57:33.781508 21611 leader_election.cc:304] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters:
I20251026 14:57:33.782163 21874 raft_consensus.cc:2804] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:57:33.782541 21874 raft_consensus.cc:697] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 2 LEADER]: Becoming Leader. State: Replica: 21eada14be174e129814165b15454da2, State: Running, Role: LEADER
I20251026 14:57:33.783245 21874 consensus_queue.cc:237] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.785320 21795 raft_consensus.cc:2468] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21eada14be174e129814165b15454da2 in term 2.
I20251026 14:57:33.788996 21908 raft_consensus.cc:993] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2: : Instructing follower d2d4f51a45dc439ea67c2fe81ed6b1a5 to start an election
I20251026 14:57:33.789480 21908 raft_consensus.cc:1081] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [term 1 LEADER]: Signalling peer d2d4f51a45dc439ea67c2fe81ed6b1a5 to start an election
I20251026 14:57:33.791172 21722 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0e31292ae87b4a04a394a31ca556654b"
dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
from {username='slave'} at 127.0.0.1:35966
I20251026 14:57:33.791741 21722 raft_consensus.cc:493] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:57:33.792194 21722 raft_consensus.cc:3060] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:33.791831 21543 catalog_manager.cc:5649] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 reported cstate change: term changed from 1 to 2, leader changed from cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3) to 21eada14be174e129814165b15454da2 (127.21.2.1). New cstate: current_term: 2 leader_uuid: "21eada14be174e129814165b15454da2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: UNKNOWN } } }
I20251026 14:57:33.798923 21722 raft_consensus.cc:515] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.801755 21646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e31292ae87b4a04a394a31ca556654b" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "21eada14be174e129814165b15454da2"
I20251026 14:57:33.802321 21646 raft_consensus.cc:3055] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:57:33.802655 21646 raft_consensus.cc:740] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 21eada14be174e129814165b15454da2, State: Running, Role: LEADER
I20251026 14:57:33.803416 21646 consensus_queue.cc:260] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.803692 21795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e31292ae87b4a04a394a31ca556654b" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
I20251026 14:57:33.804304 21795 raft_consensus.cc:3060] T 0e31292ae87b4a04a394a31ca556654b P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:33.804418 21722 leader_election.cc:290] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 2 election: Requested vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457)
I20251026 14:57:33.804327 21646 raft_consensus.cc:3060] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:33.809783 21646 raft_consensus.cc:2468] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 2.
I20251026 14:57:33.810775 21686 leader_election.cc:304] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters:
I20251026 14:57:33.811239 21795 raft_consensus.cc:2468] T 0e31292ae87b4a04a394a31ca556654b P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 2.
I20251026 14:57:33.811455 21849 raft_consensus.cc:2804] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:57:33.811964 21849 raft_consensus.cc:697] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 LEADER]: Becoming Leader. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Running, Role: LEADER
I20251026 14:57:33.812798 21849 consensus_queue.cc:237] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:33.818958 21543 catalog_manager.cc:5649] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5 reported cstate change: term changed from 1 to 2, leader changed from 21eada14be174e129814165b15454da2 (127.21.2.1) to d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2). New cstate: current_term: 2 leader_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: UNKNOWN } } }
I20251026 14:57:34.032872 21646 raft_consensus.cc:1275] T d4910c8c220b4256913e4ab10a75ebe1 P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Refusing update from remote peer cdc74a031aba4e589df4c9eb8590cc57: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:57:34.034195 21845 consensus_queue.cc:1048] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:57:34.048770 21722 raft_consensus.cc:1275] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Refusing update from remote peer cdc74a031aba4e589df4c9eb8590cc57: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:57:34.050349 21863 consensus_queue.cc:1048] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:57:34.056740 21795 raft_consensus.cc:1275] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Refusing update from remote peer d2d4f51a45dc439ea67c2fe81ed6b1a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:57:34.058468 21849 consensus_queue.cc:1048] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:57:34.072032 21646 raft_consensus.cc:1275] T 90496135145648aba3eb19b0190c35c6 P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Refusing update from remote peer d2d4f51a45dc439ea67c2fe81ed6b1a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:57:34.072289 21796 raft_consensus.cc:1275] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Refusing update from remote peer 21eada14be174e129814165b15454da2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:57:34.073792 21874 consensus_queue.cc:1048] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:57:34.074527 21849 consensus_queue.cc:1048] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:57:34.091200 21722 raft_consensus.cc:1275] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Refusing update from remote peer 21eada14be174e129814165b15454da2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:57:34.092857 21874 consensus_queue.cc:1048] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:57:34.248553 21795 raft_consensus.cc:1275] T 0e31292ae87b4a04a394a31ca556654b P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Refusing update from remote peer d2d4f51a45dc439ea67c2fe81ed6b1a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:57:34.248553 21646 raft_consensus.cc:1275] T 1ecb97a63a3c422eb2079ce004a1c7ec P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Refusing update from remote peer d2d4f51a45dc439ea67c2fe81ed6b1a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:57:34.250108 21849 consensus_queue.cc:1048] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:57:34.250033 21919 consensus_queue.cc:1048] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:57:34.256827 21796 raft_consensus.cc:1275] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Refusing update from remote peer d2d4f51a45dc439ea67c2fe81ed6b1a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:57:34.265064 21925 consensus_queue.cc:1048] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:57:34.271102 21645 raft_consensus.cc:1275] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Refusing update from remote peer d2d4f51a45dc439ea67c2fe81ed6b1a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:57:34.273051 21926 consensus_queue.cc:1048] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:57:34.308400 21795 raft_consensus.cc:1275] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Refusing update from remote peer 21eada14be174e129814165b15454da2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:57:34.309608 21874 consensus_queue.cc:1048] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:57:34.316821 21722 raft_consensus.cc:1275] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Refusing update from remote peer 21eada14be174e129814165b15454da2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:57:34.318396 21908 consensus_queue.cc:1048] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251026 14:57:35.874570 21817 debug-util.cc:398] Leaking SignalData structure 0x7b080016d060 after lost signal to thread 21515
W20251026 14:57:35.875312 21817 debug-util.cc:398] Leaking SignalData structure 0x7b080016d740 after lost signal to thread 21578
W20251026 14:57:35.876085 21817 debug-util.cc:398] Leaking SignalData structure 0x7b08000c3580 after lost signal to thread 21671
W20251026 14:57:35.876758 21817 debug-util.cc:398] Leaking SignalData structure 0x7b08001dc3c0 after lost signal to thread 21746
W20251026 14:57:35.877429 21817 debug-util.cc:398] Leaking SignalData structure 0x7b080014e7a0 after lost signal to thread 21820
I20251026 14:57:36.776810 21512 auto_leader_rebalancer-test.cc:272] The leader distribution is
I20251026 14:57:36.778163 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:57:36.800318 21646 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "859bffc81a9b4a248c1b284e6bc3d41b"
dest_uuid: "21eada14be174e129814165b15454da2"
mode: GRACEFUL
new_leader_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
from {username='slave'} at 127.0.0.1:37396
I20251026 14:57:36.800907 21646 raft_consensus.cc:606] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 2 LEADER]: Received request to transfer leadership to TS cdc74a031aba4e589df4c9eb8590cc57
I20251026 14:57:36.814078 21718 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "90496135145648aba3eb19b0190c35c6"
dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
mode: GRACEFUL
new_leader_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
from {username='slave'} at 127.0.0.1:35986
I20251026 14:57:36.814750 21718 raft_consensus.cc:606] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 LEADER]: Received request to transfer leadership to TS cdc74a031aba4e589df4c9eb8590cc57
I20251026 14:57:36.815923 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251026 14:57:36.816573 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:57:36.842972 21908 raft_consensus.cc:993] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2: : Instructing follower cdc74a031aba4e589df4c9eb8590cc57 to start an election
I20251026 14:57:36.843511 21934 raft_consensus.cc:1081] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 2 LEADER]: Signalling peer cdc74a031aba4e589df4c9eb8590cc57 to start an election
I20251026 14:57:36.845116 21794 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "859bffc81a9b4a248c1b284e6bc3d41b"
dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
from {username='slave'} at 127.0.0.1:48110
I20251026 14:57:36.845649 21794 raft_consensus.cc:493] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:57:36.845925 21794 raft_consensus.cc:3060] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:57:36.850683 21794 raft_consensus.cc:515] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } }
I20251026 14:57:36.852391 21794 leader_election.cc:290] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 3 election: Requested vote from peers d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377), 21eada14be174e129814165b15454da2 (127.21.2.1:32983)
I20251026 14:57:36.853636 21646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "859bffc81a9b4a248c1b284e6bc3d41b" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "21eada14be174e129814165b15454da2"
I20251026 14:57:36.854297 21646 raft_consensus.cc:3055] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:57:36.854295 21718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "859bffc81a9b4a248c1b284e6bc3d41b" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
I20251026 14:57:36.854794 21646 raft_consensus.cc:740] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 21eada14be174e129814165b15454da2, State: Running, Role: LEADER
I20251026 14:57:36.854957 21718 raft_consensus.cc:3060] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:57:36.855670 21646 consensus_queue.cc:260] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } }
I20251026 14:57:36.857023 21646 raft_consensus.cc:3060] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:57:36.859937 21718 raft_consensus.cc:2468] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc74a031aba4e589df4c9eb8590cc57 in term 3.
I20251026 14:57:36.861078 21760 leader_election.cc:304] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdc74a031aba4e589df4c9eb8590cc57, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters:
I20251026 14:57:36.861799 21863 raft_consensus.cc:2804] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:57:36.863088 21863 raft_consensus.cc:697] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 3 LEADER]: Becoming Leader. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Running, Role: LEADER
I20251026 14:57:36.863874 21863 consensus_queue.cc:237] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } }
I20251026 14:57:36.865841 21646 raft_consensus.cc:2468] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc74a031aba4e589df4c9eb8590cc57 in term 3.
I20251026 14:57:36.874080 21544 catalog_manager.cc:5649] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 reported cstate change: term changed from 2 to 3, leader changed from 21eada14be174e129814165b15454da2 (127.21.2.1) to cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3). New cstate: current_term: 3 leader_uuid: "cdc74a031aba4e589df4c9eb8590cc57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: UNKNOWN } } }
I20251026 14:57:37.008252 21944 raft_consensus.cc:993] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5: : Instructing follower cdc74a031aba4e589df4c9eb8590cc57 to start an election
I20251026 14:57:37.009028 21944 raft_consensus.cc:1081] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 LEADER]: Signalling peer cdc74a031aba4e589df4c9eb8590cc57 to start an election
I20251026 14:57:37.010845 21794 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "90496135145648aba3eb19b0190c35c6"
dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
from {username='slave'} at 127.0.0.1:48104
I20251026 14:57:37.011555 21794 raft_consensus.cc:493] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:57:37.012012 21794 raft_consensus.cc:3060] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:57:37.019759 21794 raft_consensus.cc:515] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:37.021994 21794 leader_election.cc:290] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 3 election: Requested vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377)
I20251026 14:57:37.022874 21646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90496135145648aba3eb19b0190c35c6" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "21eada14be174e129814165b15454da2"
I20251026 14:57:37.023133 21722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90496135145648aba3eb19b0190c35c6" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
I20251026 14:57:37.023494 21646 raft_consensus.cc:3060] T 90496135145648aba3eb19b0190c35c6 P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:57:37.023643 21722 raft_consensus.cc:3055] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:57:37.024003 21722 raft_consensus.cc:740] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Running, Role: LEADER
I20251026 14:57:37.024873 21722 consensus_queue.cc:260] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:37.026253 21722 raft_consensus.cc:3060] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:57:37.031024 21646 raft_consensus.cc:2468] T 90496135145648aba3eb19b0190c35c6 P 21eada14be174e129814165b15454da2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc74a031aba4e589df4c9eb8590cc57 in term 3.
I20251026 14:57:37.032372 21760 leader_election.cc:304] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, cdc74a031aba4e589df4c9eb8590cc57; no voters:
I20251026 14:57:37.033353 21863 raft_consensus.cc:2804] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:57:37.033823 21863 raft_consensus.cc:697] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 3 LEADER]: Becoming Leader. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Running, Role: LEADER
I20251026 14:57:37.034550 21722 raft_consensus.cc:2468] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc74a031aba4e589df4c9eb8590cc57 in term 3.
I20251026 14:57:37.034664 21863 consensus_queue.cc:237] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:37.044272 21544 catalog_manager.cc:5649] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 reported cstate change: term changed from 2 to 3, leader changed from d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2) to cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3). New cstate: current_term: 3 leader_uuid: "cdc74a031aba4e589df4c9eb8590cc57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: HEALTHY } } }
I20251026 14:57:37.263548 21646 raft_consensus.cc:1275] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 3 FOLLOWER]: Refusing update from remote peer cdc74a031aba4e589df4c9eb8590cc57: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:57:37.264784 21863 consensus_queue.cc:1048] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:57:37.282677 21722 raft_consensus.cc:1275] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 FOLLOWER]: Refusing update from remote peer cdc74a031aba4e589df4c9eb8590cc57: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:57:37.284538 21938 consensus_queue.cc:1048] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:57:37.464208 21722 raft_consensus.cc:1275] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 FOLLOWER]: Refusing update from remote peer cdc74a031aba4e589df4c9eb8590cc57: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:57:37.465595 21938 consensus_queue.cc:1048] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:57:37.475483 21646 raft_consensus.cc:1275] T 90496135145648aba3eb19b0190c35c6 P 21eada14be174e129814165b15454da2 [term 3 FOLLOWER]: Refusing update from remote peer cdc74a031aba4e589df4c9eb8590cc57: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:57:37.477118 21863 consensus_queue.cc:1048] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:57:37.817768 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:57:37.825978 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:57:37.826400 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:57:38.827281 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:57:38.832202 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:57:38.832639 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:57:39.833897 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:57:39.839783 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:57:39.840308 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:57:40.841445 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:57:40.846799 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:57:40.847239 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:57:41.848086 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:57:41.852901 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:57:41.853404 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:57:42.854380 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:57:42.859694 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:57:42.860167 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:57:43.861639 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:57:43.866443 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:57:43.866871 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:57:44.867806 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:57:44.872948 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:57:44.873440 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:57:45.874316 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:57:45.879042 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:57:45.879487 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:57:46.908955 21512 auto_leader_rebalancer-test.cc:291] The leader distribution is
I20251026 14:57:46.913990 21796 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "859bffc81a9b4a248c1b284e6bc3d41b"
dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
mode: GRACEFUL
new_leader_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
from {username='slave'} at 127.0.0.1:48116
I20251026 14:57:46.915002 21796 raft_consensus.cc:606] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 3 LEADER]: Received request to transfer leadership to TS d2d4f51a45dc439ea67c2fe81ed6b1a5
I20251026 14:57:46.920159 21646 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d12e45731a5146be91de773fcc96acf4"
dest_uuid: "21eada14be174e129814165b15454da2"
mode: GRACEFUL
new_leader_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
from {username='slave'} at 127.0.0.1:44010
I20251026 14:57:46.920617 21646 raft_consensus.cc:606] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 1 LEADER]: Received request to transfer leadership to TS d2d4f51a45dc439ea67c2fe81ed6b1a5
I20251026 14:57:46.925169 21646 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2338d7ddd3a54537a9fbc8f7fc46de45"
dest_uuid: "21eada14be174e129814165b15454da2"
mode: GRACEFUL
new_leader_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
from {username='slave'} at 127.0.0.1:44010
I20251026 14:57:46.925664 21646 raft_consensus.cc:606] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 2 LEADER]: Received request to transfer leadership to TS d2d4f51a45dc439ea67c2fe81ed6b1a5
I20251026 14:57:46.929999 21646 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "fc1d64845a9543119c22b77b1256f9f3"
dest_uuid: "21eada14be174e129814165b15454da2"
mode: GRACEFUL
new_leader_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
from {username='slave'} at 127.0.0.1:44010
I20251026 14:57:46.930449 21646 raft_consensus.cc:606] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 1 LEADER]: Received request to transfer leadership to TS d2d4f51a45dc439ea67c2fe81ed6b1a5
I20251026 14:57:46.941679 21796 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "90496135145648aba3eb19b0190c35c6"
dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
mode: GRACEFUL
new_leader_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
from {username='slave'} at 127.0.0.1:48116
I20251026 14:57:46.942159 21796 raft_consensus.cc:606] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 3 LEADER]: Received request to transfer leadership to TS d2d4f51a45dc439ea67c2fe81ed6b1a5
I20251026 14:57:47.114323 21961 raft_consensus.cc:993] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57: : Instructing follower d2d4f51a45dc439ea67c2fe81ed6b1a5 to start an election
I20251026 14:57:47.114748 21961 raft_consensus.cc:1081] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 3 LEADER]: Signalling peer d2d4f51a45dc439ea67c2fe81ed6b1a5 to start an election
I20251026 14:57:47.116214 21718 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "90496135145648aba3eb19b0190c35c6"
dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
from {username='slave'} at 127.0.0.1:35952
I20251026 14:57:47.116624 21718 raft_consensus.cc:493] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:57:47.116904 21718 raft_consensus.cc:3060] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:57:47.121454 21718 raft_consensus.cc:515] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:47.123138 21718 leader_election.cc:290] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 4 election: Requested vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457)
I20251026 14:57:47.124199 21646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90496135145648aba3eb19b0190c35c6" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "21eada14be174e129814165b15454da2"
I20251026 14:57:47.124271 21796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90496135145648aba3eb19b0190c35c6" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
I20251026 14:57:47.124881 21646 raft_consensus.cc:3060] T 90496135145648aba3eb19b0190c35c6 P 21eada14be174e129814165b15454da2 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:57:47.124966 21796 raft_consensus.cc:3055] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:57:47.125315 21796 raft_consensus.cc:740] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Running, Role: LEADER
I20251026 14:57:47.125939 21796 consensus_queue.cc:260] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:47.126823 21796 raft_consensus.cc:3060] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:57:47.129238 21646 raft_consensus.cc:2468] T 90496135145648aba3eb19b0190c35c6 P 21eada14be174e129814165b15454da2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 4.
I20251026 14:57:47.130465 21686 leader_election.cc:304] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters:
I20251026 14:57:47.131389 21955 raft_consensus.cc:2804] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:57:47.131536 21796 raft_consensus.cc:2468] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 4.
I20251026 14:57:47.132398 21955 raft_consensus.cc:697] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 4 LEADER]: Becoming Leader. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Running, Role: LEADER
I20251026 14:57:47.133350 21955 consensus_queue.cc:237] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:47.140594 21543 catalog_manager.cc:5649] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 reported cstate change: term changed from 3 to 4, leader changed from cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3) to d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2). New cstate: current_term: 4 leader_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: UNKNOWN } } }
I20251026 14:57:47.181846 21934 raft_consensus.cc:993] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2: : Instructing follower d2d4f51a45dc439ea67c2fe81ed6b1a5 to start an election
I20251026 14:57:47.182381 21969 raft_consensus.cc:1081] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 2 LEADER]: Signalling peer d2d4f51a45dc439ea67c2fe81ed6b1a5 to start an election
I20251026 14:57:47.183763 21718 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2338d7ddd3a54537a9fbc8f7fc46de45"
dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
from {username='slave'} at 127.0.0.1:35966
I20251026 14:57:47.184197 21718 raft_consensus.cc:493] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:57:47.184460 21718 raft_consensus.cc:3060] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:57:47.188969 21934 raft_consensus.cc:993] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2: : Instructing follower d2d4f51a45dc439ea67c2fe81ed6b1a5 to start an election
I20251026 14:57:47.189404 21969 raft_consensus.cc:1081] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 1 LEADER]: Signalling peer d2d4f51a45dc439ea67c2fe81ed6b1a5 to start an election
I20251026 14:57:47.189908 21718 raft_consensus.cc:515] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:47.191195 21722 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "fc1d64845a9543119c22b77b1256f9f3"
dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
from {username='slave'} at 127.0.0.1:35966
I20251026 14:57:47.191583 21718 leader_election.cc:290] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 3 election: Requested vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457)
I20251026 14:57:47.191821 21722 raft_consensus.cc:493] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:57:47.192286 21646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2338d7ddd3a54537a9fbc8f7fc46de45" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "21eada14be174e129814165b15454da2"
I20251026 14:57:47.193894 21796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2338d7ddd3a54537a9fbc8f7fc46de45" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
I20251026 14:57:47.196692 21722 raft_consensus.cc:3060] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:47.196981 21646 raft_consensus.cc:3055] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:57:47.197052 21796 raft_consensus.cc:3060] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:57:47.197300 21646 raft_consensus.cc:740] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 21eada14be174e129814165b15454da2, State: Running, Role: LEADER
I20251026 14:57:47.197988 21646 consensus_queue.cc:260] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:47.198869 21646 raft_consensus.cc:3060] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:57:47.203182 21646 raft_consensus.cc:2468] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 3.
I20251026 14:57:47.203269 21796 raft_consensus.cc:2468] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 3.
I20251026 14:57:47.203298 21722 raft_consensus.cc:515] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:47.204342 21686 leader_election.cc:304] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters:
I20251026 14:57:47.205294 21955 raft_consensus.cc:2804] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:57:47.205772 21955 raft_consensus.cc:697] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 LEADER]: Becoming Leader. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Running, Role: LEADER
I20251026 14:57:47.206413 21722 leader_election.cc:290] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 2 election: Requested vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457)
I20251026 14:57:47.206983 21796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc1d64845a9543119c22b77b1256f9f3" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
I20251026 14:57:47.206668 21955 consensus_queue.cc:237] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:47.207631 21796 raft_consensus.cc:3060] T fc1d64845a9543119c22b77b1256f9f3 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:47.206673 21646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc1d64845a9543119c22b77b1256f9f3" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "21eada14be174e129814165b15454da2"
I20251026 14:57:47.209829 21646 raft_consensus.cc:3055] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:57:47.210134 21646 raft_consensus.cc:740] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 21eada14be174e129814165b15454da2, State: Running, Role: LEADER
I20251026 14:57:47.210585 21961 raft_consensus.cc:993] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57: : Instructing follower d2d4f51a45dc439ea67c2fe81ed6b1a5 to start an election
I20251026 14:57:47.210996 21961 raft_consensus.cc:1081] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 3 LEADER]: Signalling peer d2d4f51a45dc439ea67c2fe81ed6b1a5 to start an election
I20251026 14:57:47.210922 21646 consensus_queue.cc:260] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:47.212121 21646 raft_consensus.cc:3060] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:47.212728 21718 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "859bffc81a9b4a248c1b284e6bc3d41b"
dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
from {username='slave'} at 127.0.0.1:35952
I20251026 14:57:47.213227 21718 raft_consensus.cc:493] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:57:47.213554 21718 raft_consensus.cc:3060] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:57:47.216159 21796 raft_consensus.cc:2468] T fc1d64845a9543119c22b77b1256f9f3 P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 2.
I20251026 14:57:47.217645 21687 leader_election.cc:304] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdc74a031aba4e589df4c9eb8590cc57, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters:
I20251026 14:57:47.218919 21963 raft_consensus.cc:2804] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:57:47.220413 21963 raft_consensus.cc:697] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 LEADER]: Becoming Leader. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Running, Role: LEADER
I20251026 14:57:47.221513 21963 consensus_queue.cc:237] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:47.217775 21718 raft_consensus.cc:515] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } }
I20251026 14:57:47.226346 21543 catalog_manager.cc:5649] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 reported cstate change: term changed from 2 to 3, leader changed from 21eada14be174e129814165b15454da2 (127.21.2.1) to d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2). New cstate: current_term: 3 leader_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: UNKNOWN } } }
I20251026 14:57:47.227120 21796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "859bffc81a9b4a248c1b284e6bc3d41b" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
I20251026 14:57:47.227233 21645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "859bffc81a9b4a248c1b284e6bc3d41b" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "21eada14be174e129814165b15454da2"
I20251026 14:57:47.227758 21796 raft_consensus.cc:3055] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:57:47.227840 21718 leader_election.cc:290] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 4 election: Requested vote from peers cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457), 21eada14be174e129814165b15454da2 (127.21.2.1:32983)
I20251026 14:57:47.227778 21645 raft_consensus.cc:3060] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:57:47.228166 21796 raft_consensus.cc:740] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Running, Role: LEADER
I20251026 14:57:47.228950 21796 consensus_queue.cc:260] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } }
I20251026 14:57:47.229745 21646 raft_consensus.cc:2468] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 2.
I20251026 14:57:47.230159 21796 raft_consensus.cc:3060] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:57:47.233959 21645 raft_consensus.cc:2468] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 4.
I20251026 14:57:47.234939 21686 leader_election.cc:304] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters:
I20251026 14:57:47.235888 21796 raft_consensus.cc:2468] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 4.
I20251026 14:57:47.236629 21963 raft_consensus.cc:2804] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:57:47.237095 21963 raft_consensus.cc:697] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 4 LEADER]: Becoming Leader. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Running, Role: LEADER
I20251026 14:57:47.237869 21963 consensus_queue.cc:237] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } }
I20251026 14:57:47.245182 21543 catalog_manager.cc:5649] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 reported cstate change: term changed from 1 to 2, leader changed from 21eada14be174e129814165b15454da2 (127.21.2.1) to d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2). New cstate: current_term: 2 leader_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: UNKNOWN } } }
I20251026 14:57:47.246413 21543 catalog_manager.cc:5649] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 reported cstate change: term changed from 3 to 4, leader changed from cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3) to d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2). New cstate: current_term: 4 leader_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: UNKNOWN } } }
I20251026 14:57:47.284166 21934 raft_consensus.cc:993] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2: : Instructing follower d2d4f51a45dc439ea67c2fe81ed6b1a5 to start an election
I20251026 14:57:47.284535 21969 raft_consensus.cc:1081] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 1 LEADER]: Signalling peer d2d4f51a45dc439ea67c2fe81ed6b1a5 to start an election
I20251026 14:57:47.285895 21718 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "d12e45731a5146be91de773fcc96acf4"
dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
from {username='slave'} at 127.0.0.1:35966
I20251026 14:57:47.286343 21718 raft_consensus.cc:493] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:57:47.286594 21718 raft_consensus.cc:3060] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:47.290546 21718 raft_consensus.cc:515] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:47.291816 21718 leader_election.cc:290] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 2 election: Requested vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457)
I20251026 14:57:47.292821 21645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d12e45731a5146be91de773fcc96acf4" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "21eada14be174e129814165b15454da2"
I20251026 14:57:47.292799 21796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d12e45731a5146be91de773fcc96acf4" candidate_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
I20251026 14:57:47.293556 21796 raft_consensus.cc:3060] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:47.293519 21645 raft_consensus.cc:3055] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:57:47.294023 21645 raft_consensus.cc:740] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 21eada14be174e129814165b15454da2, State: Running, Role: LEADER
I20251026 14:57:47.294785 21645 consensus_queue.cc:260] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:47.296046 21645 raft_consensus.cc:3060] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:47.300197 21796 raft_consensus.cc:2468] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 2.
I20251026 14:57:47.301353 21687 leader_election.cc:304] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdc74a031aba4e589df4c9eb8590cc57, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters:
I20251026 14:57:47.302039 21963 raft_consensus.cc:2804] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:57:47.302350 21645 raft_consensus.cc:2468] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d4f51a45dc439ea67c2fe81ed6b1a5 in term 2.
I20251026 14:57:47.302407 21963 raft_consensus.cc:697] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 LEADER]: Becoming Leader. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Running, Role: LEADER
I20251026 14:57:47.303396 21963 consensus_queue.cc:237] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:47.309684 21543 catalog_manager.cc:5649] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 reported cstate change: term changed from 1 to 2, leader changed from 21eada14be174e129814165b15454da2 (127.21.2.1) to d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2). New cstate: current_term: 2 leader_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: UNKNOWN } } }
I20251026 14:57:47.664163 21796 raft_consensus.cc:1275] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 3 FOLLOWER]: Refusing update from remote peer d2d4f51a45dc439ea67c2fe81ed6b1a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:57:47.665365 21963 consensus_queue.cc:1048] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:57:47.669711 21794 raft_consensus.cc:1275] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 4 FOLLOWER]: Refusing update from remote peer d2d4f51a45dc439ea67c2fe81ed6b1a5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251026 14:57:47.671203 21963 consensus_queue.cc:1048] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:57:47.682008 21645 raft_consensus.cc:1275] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 3 FOLLOWER]: Refusing update from remote peer d2d4f51a45dc439ea67c2fe81ed6b1a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:57:47.683565 21963 consensus_queue.cc:1048] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:57:47.684834 21645 raft_consensus.cc:1275] T 90496135145648aba3eb19b0190c35c6 P 21eada14be174e129814165b15454da2 [term 4 FOLLOWER]: Refusing update from remote peer d2d4f51a45dc439ea67c2fe81ed6b1a5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251026 14:57:47.687758 21955 consensus_queue.cc:1048] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251026 14:57:47.698333 21794 raft_consensus.cc:1275] T fc1d64845a9543119c22b77b1256f9f3 P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Refusing update from remote peer d2d4f51a45dc439ea67c2fe81ed6b1a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:57:47.700623 21979 consensus_queue.cc:1048] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:57:47.719451 21646 raft_consensus.cc:1275] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Refusing update from remote peer d2d4f51a45dc439ea67c2fe81ed6b1a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:57:47.722486 21963 consensus_queue.cc:1048] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:57:47.772608 21646 raft_consensus.cc:1275] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 4 FOLLOWER]: Refusing update from remote peer d2d4f51a45dc439ea67c2fe81ed6b1a5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251026 14:57:47.773825 21963 consensus_queue.cc:1048] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:57:47.782289 21794 raft_consensus.cc:1275] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 4 FOLLOWER]: Refusing update from remote peer d2d4f51a45dc439ea67c2fe81ed6b1a5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251026 14:57:47.784925 21963 consensus_queue.cc:1048] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:57:47.801877 21794 raft_consensus.cc:1275] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Refusing update from remote peer d2d4f51a45dc439ea67c2fe81ed6b1a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:57:47.803164 21963 consensus_queue.cc:1048] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:57:47.813376 21646 raft_consensus.cc:1275] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Refusing update from remote peer d2d4f51a45dc439ea67c2fe81ed6b1a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:57:47.815101 21979 consensus_queue.cc:1048] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:57:49.975613 21512 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20251026 14:57:49.976480 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:57:49.983579 21718 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2338d7ddd3a54537a9fbc8f7fc46de45"
dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
mode: GRACEFUL
new_leader_uuid: "21eada14be174e129814165b15454da2"
from {username='slave'} at 127.0.0.1:35986
I20251026 14:57:49.984408 21718 raft_consensus.cc:606] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 LEADER]: Received request to transfer leadership to TS 21eada14be174e129814165b15454da2
I20251026 14:57:49.987205 21718 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "859bffc81a9b4a248c1b284e6bc3d41b"
dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
mode: GRACEFUL
new_leader_uuid: "21eada14be174e129814165b15454da2"
from {username='slave'} at 127.0.0.1:35986
I20251026 14:57:49.987623 21718 raft_consensus.cc:606] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 4 LEADER]: Received request to transfer leadership to TS 21eada14be174e129814165b15454da2
I20251026 14:57:49.989704 21718 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d12e45731a5146be91de773fcc96acf4"
dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
mode: GRACEFUL
new_leader_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
from {username='slave'} at 127.0.0.1:35986
I20251026 14:57:49.990170 21718 raft_consensus.cc:606] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 LEADER]: Received request to transfer leadership to TS cdc74a031aba4e589df4c9eb8590cc57
I20251026 14:57:49.992787 21718 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d676fec08f2a45b8b44e1e1574606e6a"
dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
mode: GRACEFUL
new_leader_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
from {username='slave'} at 127.0.0.1:35986
I20251026 14:57:49.993314 21718 raft_consensus.cc:606] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 LEADER]: Received request to transfer leadership to TS cdc74a031aba4e589df4c9eb8590cc57
I20251026 14:57:49.995822 21718 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "fc1d64845a9543119c22b77b1256f9f3"
dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
mode: GRACEFUL
new_leader_uuid: "21eada14be174e129814165b15454da2"
from {username='slave'} at 127.0.0.1:35986
I20251026 14:57:49.996385 21718 raft_consensus.cc:606] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 LEADER]: Received request to transfer leadership to TS 21eada14be174e129814165b15454da2
I20251026 14:57:49.997466 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251026 14:57:49.998020 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:57:50.080529 21991 raft_consensus.cc:993] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5: : Instructing follower 21eada14be174e129814165b15454da2 to start an election
I20251026 14:57:50.081087 21991 raft_consensus.cc:1081] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 LEADER]: Signalling peer 21eada14be174e129814165b15454da2 to start an election
I20251026 14:57:50.082615 21645 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2338d7ddd3a54537a9fbc8f7fc46de45"
dest_uuid: "21eada14be174e129814165b15454da2"
from {username='slave'} at 127.0.0.1:43998
I20251026 14:57:50.083089 21645 raft_consensus.cc:493] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:57:50.083407 21645 raft_consensus.cc:3060] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:57:50.089031 21645 raft_consensus.cc:515] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:50.091276 21645 leader_election.cc:290] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [CANDIDATE]: Term 4 election: Requested vote from peers d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377), cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457)
I20251026 14:57:50.092638 21794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2338d7ddd3a54537a9fbc8f7fc46de45" candidate_uuid: "21eada14be174e129814165b15454da2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
I20251026 14:57:50.092373 21718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2338d7ddd3a54537a9fbc8f7fc46de45" candidate_uuid: "21eada14be174e129814165b15454da2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
I20251026 14:57:50.093391 21794 raft_consensus.cc:3060] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:57:50.093457 21718 raft_consensus.cc:3055] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:57:50.093839 21718 raft_consensus.cc:740] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Running, Role: LEADER
I20251026 14:57:50.094501 21718 consensus_queue.cc:260] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:50.095485 21718 raft_consensus.cc:3060] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:57:50.099248 21794 raft_consensus.cc:2468] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21eada14be174e129814165b15454da2 in term 4.
I20251026 14:57:50.100126 21718 raft_consensus.cc:2468] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21eada14be174e129814165b15454da2 in term 4.
I20251026 14:57:50.100618 21612 leader_election.cc:304] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, cdc74a031aba4e589df4c9eb8590cc57; no voters:
I20251026 14:57:50.102267 21993 raft_consensus.cc:2804] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:57:50.103137 21993 raft_consensus.cc:697] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 4 LEADER]: Becoming Leader. State: Replica: 21eada14be174e129814165b15454da2, State: Running, Role: LEADER
I20251026 14:57:50.104049 21993 consensus_queue.cc:237] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:50.111111 21543 catalog_manager.cc:5649] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 reported cstate change: term changed from 3 to 4, leader changed from d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2) to 21eada14be174e129814165b15454da2 (127.21.2.1). New cstate: current_term: 4 leader_uuid: "21eada14be174e129814165b15454da2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: UNKNOWN } } }
I20251026 14:57:50.235786 21991 raft_consensus.cc:993] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5: : Instructing follower 21eada14be174e129814165b15454da2 to start an election
I20251026 14:57:50.236434 21955 raft_consensus.cc:1081] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 LEADER]: Signalling peer 21eada14be174e129814165b15454da2 to start an election
I20251026 14:57:50.237825 21645 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "fc1d64845a9543119c22b77b1256f9f3"
dest_uuid: "21eada14be174e129814165b15454da2"
from {username='slave'} at 127.0.0.1:43998
I20251026 14:57:50.238296 21645 raft_consensus.cc:493] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:57:50.238670 21645 raft_consensus.cc:3060] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:57:50.239449 21992 raft_consensus.cc:993] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5: : Instructing follower cdc74a031aba4e589df4c9eb8590cc57 to start an election
I20251026 14:57:50.239873 21955 raft_consensus.cc:1081] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 LEADER]: Signalling peer cdc74a031aba4e589df4c9eb8590cc57 to start an election
I20251026 14:57:50.241518 21796 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "d676fec08f2a45b8b44e1e1574606e6a"
dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
from {username='slave'} at 127.0.0.1:48104
I20251026 14:57:50.241957 21796 raft_consensus.cc:493] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:57:50.242202 21796 raft_consensus.cc:3060] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:50.244697 21645 raft_consensus.cc:515] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:50.246315 21645 leader_election.cc:290] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [CANDIDATE]: Term 3 election: Requested vote from peers d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377), cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457)
I20251026 14:57:50.246698 21796 raft_consensus.cc:515] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:50.247130 21718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc1d64845a9543119c22b77b1256f9f3" candidate_uuid: "21eada14be174e129814165b15454da2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
I20251026 14:57:50.247378 21794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc1d64845a9543119c22b77b1256f9f3" candidate_uuid: "21eada14be174e129814165b15454da2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
I20251026 14:57:50.247694 21718 raft_consensus.cc:3055] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:57:50.247917 21794 raft_consensus.cc:3060] T fc1d64845a9543119c22b77b1256f9f3 P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:57:50.248026 21718 raft_consensus.cc:740] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Running, Role: LEADER
I20251026 14:57:50.249094 21796 leader_election.cc:290] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 2 election: Requested vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377)
I20251026 14:57:50.248798 21718 consensus_queue.cc:260] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:50.250130 21718 raft_consensus.cc:3060] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:57:50.250676 21722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d676fec08f2a45b8b44e1e1574606e6a" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
I20251026 14:57:50.251121 21722 raft_consensus.cc:3055] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:57:50.251330 21722 raft_consensus.cc:740] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Running, Role: LEADER
I20251026 14:57:50.251919 21722 consensus_queue.cc:260] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:50.252681 21722 raft_consensus.cc:3060] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:50.254918 21794 raft_consensus.cc:2468] T fc1d64845a9543119c22b77b1256f9f3 P cdc74a031aba4e589df4c9eb8590cc57 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21eada14be174e129814165b15454da2 in term 3.
I20251026 14:57:50.255973 21645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d676fec08f2a45b8b44e1e1574606e6a" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "21eada14be174e129814165b15454da2"
I20251026 14:57:50.256179 21612 leader_election.cc:304] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, cdc74a031aba4e589df4c9eb8590cc57; no voters:
I20251026 14:57:50.256543 21645 raft_consensus.cc:3060] T d676fec08f2a45b8b44e1e1574606e6a P 21eada14be174e129814165b15454da2 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:57:50.256599 21718 raft_consensus.cc:2468] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21eada14be174e129814165b15454da2 in term 3.
I20251026 14:57:50.257791 21722 raft_consensus.cc:2468] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc74a031aba4e589df4c9eb8590cc57 in term 2.
I20251026 14:57:50.259047 21993 raft_consensus.cc:2804] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:57:50.259183 21760 leader_election.cc:304] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdc74a031aba4e589df4c9eb8590cc57, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters:
I20251026 14:57:50.260121 21962 raft_consensus.cc:2804] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:57:50.260377 21993 raft_consensus.cc:697] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 3 LEADER]: Becoming Leader. State: Replica: 21eada14be174e129814165b15454da2, State: Running, Role: LEADER
I20251026 14:57:50.261582 21993 consensus_queue.cc:237] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:50.263123 21645 raft_consensus.cc:2468] T d676fec08f2a45b8b44e1e1574606e6a P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc74a031aba4e589df4c9eb8590cc57 in term 2.
I20251026 14:57:50.265424 21955 raft_consensus.cc:993] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5: : Instructing follower 21eada14be174e129814165b15454da2 to start an election
I20251026 14:57:50.265947 21955 raft_consensus.cc:1081] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 4 LEADER]: Signalling peer 21eada14be174e129814165b15454da2 to start an election
I20251026 14:57:50.265846 21962 raft_consensus.cc:697] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57 [term 2 LEADER]: Becoming Leader. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Running, Role: LEADER
I20251026 14:57:50.266916 21962 consensus_queue.cc:237] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:50.267793 21645 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "859bffc81a9b4a248c1b284e6bc3d41b"
dest_uuid: "21eada14be174e129814165b15454da2"
from {username='slave'} at 127.0.0.1:43998
I20251026 14:57:50.268415 21645 raft_consensus.cc:493] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:57:50.268775 21645 raft_consensus.cc:3060] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:57:50.271143 21543 catalog_manager.cc:5649] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 reported cstate change: term changed from 2 to 3, leader changed from d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2) to 21eada14be174e129814165b15454da2 (127.21.2.1). New cstate: current_term: 3 leader_uuid: "21eada14be174e129814165b15454da2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: UNKNOWN } } }
I20251026 14:57:50.275377 21645 raft_consensus.cc:515] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } }
I20251026 14:57:50.277897 21794 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "859bffc81a9b4a248c1b284e6bc3d41b" candidate_uuid: "21eada14be174e129814165b15454da2" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
I20251026 14:57:50.278365 21722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "859bffc81a9b4a248c1b284e6bc3d41b" candidate_uuid: "21eada14be174e129814165b15454da2" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
I20251026 14:57:50.279013 21722 raft_consensus.cc:3055] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:57:50.279320 21722 raft_consensus.cc:740] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Running, Role: LEADER
I20251026 14:57:50.278436 21645 leader_election.cc:290] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [CANDIDATE]: Term 5 election: Requested vote from peers cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3:33457), d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377)
I20251026 14:57:50.278486 21794 raft_consensus.cc:3060] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:57:50.280000 21722 consensus_queue.cc:260] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } }
I20251026 14:57:50.281066 21722 raft_consensus.cc:3060] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:57:50.276409 21542 catalog_manager.cc:5649] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57 reported cstate change: term changed from 1 to 2, leader changed from d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2) to cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3). New cstate: current_term: 2 leader_uuid: "cdc74a031aba4e589df4c9eb8590cc57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: HEALTHY } } }
I20251026 14:57:50.286439 21794 raft_consensus.cc:2468] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21eada14be174e129814165b15454da2 in term 5.
I20251026 14:57:50.287477 21722 raft_consensus.cc:2468] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21eada14be174e129814165b15454da2 in term 5.
I20251026 14:57:50.288100 21955 raft_consensus.cc:993] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5: : Instructing follower cdc74a031aba4e589df4c9eb8590cc57 to start an election
I20251026 14:57:50.288551 21955 raft_consensus.cc:1081] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 LEADER]: Signalling peer cdc74a031aba4e589df4c9eb8590cc57 to start an election
I20251026 14:57:50.288625 21611 leader_election.cc:304] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, d2d4f51a45dc439ea67c2fe81ed6b1a5; no voters:
I20251026 14:57:50.289546 21993 raft_consensus.cc:2804] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:57:50.290021 21993 raft_consensus.cc:697] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 5 LEADER]: Becoming Leader. State: Replica: 21eada14be174e129814165b15454da2, State: Running, Role: LEADER
I20251026 14:57:50.290917 21993 consensus_queue.cc:237] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } }
I20251026 14:57:50.291780 21794 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "d12e45731a5146be91de773fcc96acf4"
dest_uuid: "cdc74a031aba4e589df4c9eb8590cc57"
from {username='slave'} at 127.0.0.1:48104
I20251026 14:57:50.292340 21794 raft_consensus.cc:493] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:57:50.292732 21794 raft_consensus.cc:3060] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:57:50.298405 21794 raft_consensus.cc:515] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:50.298929 21542 catalog_manager.cc:5649] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 reported cstate change: term changed from 4 to 5, leader changed from d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2) to 21eada14be174e129814165b15454da2 (127.21.2.1). New cstate: current_term: 5 leader_uuid: "21eada14be174e129814165b15454da2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: HEALTHY } } }
I20251026 14:57:50.299993 21794 leader_election.cc:290] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 3 election: Requested vote from peers 21eada14be174e129814165b15454da2 (127.21.2.1:32983), d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2:35377)
I20251026 14:57:50.300815 21645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d12e45731a5146be91de773fcc96acf4" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "21eada14be174e129814165b15454da2"
I20251026 14:57:50.300940 21722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d12e45731a5146be91de773fcc96acf4" candidate_uuid: "cdc74a031aba4e589df4c9eb8590cc57" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5"
I20251026 14:57:50.301513 21645 raft_consensus.cc:3060] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:57:50.301625 21722 raft_consensus.cc:3055] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:57:50.302037 21722 raft_consensus.cc:740] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d2d4f51a45dc439ea67c2fe81ed6b1a5, State: Running, Role: LEADER
I20251026 14:57:50.302630 21722 consensus_queue.cc:260] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:50.303512 21722 raft_consensus.cc:3060] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:57:50.307128 21645 raft_consensus.cc:2468] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc74a031aba4e589df4c9eb8590cc57 in term 3.
I20251026 14:57:50.308377 21760 leader_election.cc:304] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eada14be174e129814165b15454da2, cdc74a031aba4e589df4c9eb8590cc57; no voters:
I20251026 14:57:50.309031 21962 raft_consensus.cc:2804] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:57:50.309516 21962 raft_consensus.cc:697] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [term 3 LEADER]: Becoming Leader. State: Replica: cdc74a031aba4e589df4c9eb8590cc57, State: Running, Role: LEADER
I20251026 14:57:50.310705 21722 raft_consensus.cc:2468] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdc74a031aba4e589df4c9eb8590cc57 in term 3.
I20251026 14:57:50.310454 21962 consensus_queue.cc:237] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } }
I20251026 14:57:50.317353 21542 catalog_manager.cc:5649] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 reported cstate change: term changed from 2 to 3, leader changed from d2d4f51a45dc439ea67c2fe81ed6b1a5 (127.21.2.2) to cdc74a031aba4e589df4c9eb8590cc57 (127.21.2.3). New cstate: current_term: 3 leader_uuid: "cdc74a031aba4e589df4c9eb8590cc57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 } health_report { overall_health: HEALTHY } } }
I20251026 14:57:50.588989 21722 raft_consensus.cc:1275] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 4 FOLLOWER]: Refusing update from remote peer 21eada14be174e129814165b15454da2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251026 14:57:50.590790 21993 consensus_queue.cc:1048] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:57:50.602259 21794 raft_consensus.cc:1275] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 4 FOLLOWER]: Refusing update from remote peer 21eada14be174e129814165b15454da2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251026 14:57:50.604727 22008 consensus_queue.cc:1048] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251026 14:57:50.678680 21645 raft_consensus.cc:1275] T d676fec08f2a45b8b44e1e1574606e6a P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Refusing update from remote peer cdc74a031aba4e589df4c9eb8590cc57: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:57:50.679883 21962 consensus_queue.cc:1048] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:57:50.683130 21794 raft_consensus.cc:1275] T fc1d64845a9543119c22b77b1256f9f3 P cdc74a031aba4e589df4c9eb8590cc57 [term 3 FOLLOWER]: Refusing update from remote peer 21eada14be174e129814165b15454da2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:57:50.684396 22008 consensus_queue.cc:1048] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:57:50.690548 21722 raft_consensus.cc:1275] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Refusing update from remote peer cdc74a031aba4e589df4c9eb8590cc57: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:57:50.692425 21962 consensus_queue.cc:1048] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:57:50.702301 21718 raft_consensus.cc:1275] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 FOLLOWER]: Refusing update from remote peer 21eada14be174e129814165b15454da2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:57:50.706204 21993 consensus_queue.cc:1048] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:57:50.784027 21794 raft_consensus.cc:1275] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 5 FOLLOWER]: Refusing update from remote peer 21eada14be174e129814165b15454da2: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251026 14:57:50.785585 21993 consensus_queue.cc:1048] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc74a031aba4e589df4c9eb8590cc57" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 14:57:50.795245 21722 raft_consensus.cc:1275] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 5 FOLLOWER]: Refusing update from remote peer 21eada14be174e129814165b15454da2: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251026 14:57:50.796483 22008 consensus_queue.cc:1048] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 14:57:50.803020 21646 raft_consensus.cc:1275] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 3 FOLLOWER]: Refusing update from remote peer cdc74a031aba4e589df4c9eb8590cc57: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:57:50.804049 21962 consensus_queue.cc:1048] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eada14be174e129814165b15454da2" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:57:50.811234 21722 raft_consensus.cc:1275] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 FOLLOWER]: Refusing update from remote peer cdc74a031aba4e589df4c9eb8590cc57: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:57:50.812497 21962 consensus_queue.cc:1048] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d4f51a45dc439ea67c2fe81ed6b1a5" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:57:50.999374 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:57:51.006639 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:57:51.007151 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:57:52.008203 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:57:52.013684 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:57:52.014222 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:57:53.015802 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:57:53.022920 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:57:53.023337 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:57:54.024340 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:57:54.029165 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:57:54.029584 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:57:55.030558 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:57:55.035395 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:57:55.035799 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:57:56.036775 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:57:56.041657 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:57:56.042073 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:57:57.043134 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:57:57.047982 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:57:57.048377 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:57:58.049443 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:57:58.055691 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:57:58.056113 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:57:59.057118 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:57:59.062064 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:57:59.062461 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:58:00.090600 21512 auto_leader_rebalancer-test.cc:318] The leader distribution is
I20251026 14:58:00.091193 21512 tablet_server.cc:178] TabletServer@127.21.2.1:0 shutting down...
I20251026 14:58:00.121621 21512 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 14:58:00.122505 21512 tablet_replica.cc:333] T d4910c8c220b4256913e4ab10a75ebe1 P 21eada14be174e129814165b15454da2: stopping tablet replica
I20251026 14:58:00.123805 21512 raft_consensus.cc:2243] T d4910c8c220b4256913e4ab10a75ebe1 P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:00.124766 21512 raft_consensus.cc:2272] T d4910c8c220b4256913e4ab10a75ebe1 P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.127768 21512 tablet_replica.cc:333] T 90496135145648aba3eb19b0190c35c6 P 21eada14be174e129814165b15454da2: stopping tablet replica
I20251026 14:58:00.128364 21512 raft_consensus.cc:2243] T 90496135145648aba3eb19b0190c35c6 P 21eada14be174e129814165b15454da2 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:00.128870 21512 raft_consensus.cc:2272] T 90496135145648aba3eb19b0190c35c6 P 21eada14be174e129814165b15454da2 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.130843 21512 tablet_replica.cc:333] T 1ecb97a63a3c422eb2079ce004a1c7ec P 21eada14be174e129814165b15454da2: stopping tablet replica
I20251026 14:58:00.131321 21512 raft_consensus.cc:2243] T 1ecb97a63a3c422eb2079ce004a1c7ec P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:00.131731 21512 raft_consensus.cc:2272] T 1ecb97a63a3c422eb2079ce004a1c7ec P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.133443 21512 tablet_replica.cc:333] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2: stopping tablet replica
I20251026 14:58:00.134112 21512 raft_consensus.cc:2243] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 3 LEADER]: Raft consensus shutting down.
I20251026 14:58:00.135082 21512 raft_consensus.cc:2272] T fc1d64845a9543119c22b77b1256f9f3 P 21eada14be174e129814165b15454da2 [term 3 FOLLOWER]: Raft consensus is shut down!
W20251026 14:58:00.135888 21760 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20251026 14:58:00.137049 21512 tablet_replica.cc:333] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2: stopping tablet replica
I20251026 14:58:00.137724 21512 raft_consensus.cc:2243] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:00.138180 21512 raft_consensus.cc:2272] T 0e31292ae87b4a04a394a31ca556654b P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Raft consensus is shut down!
W20251026 14:58:00.139299 21760 consensus_peers.cc:597] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 -> Peer 21eada14be174e129814165b15454da2 (127.21.2.1:32983): Couldn't send request to peer 21eada14be174e129814165b15454da2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:00.139957 21512 tablet_replica.cc:333] T d676fec08f2a45b8b44e1e1574606e6a P 21eada14be174e129814165b15454da2: stopping tablet replica
I20251026 14:58:00.140570 21512 raft_consensus.cc:2243] T d676fec08f2a45b8b44e1e1574606e6a P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:00.141040 21512 raft_consensus.cc:2272] T d676fec08f2a45b8b44e1e1574606e6a P 21eada14be174e129814165b15454da2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.142705 21512 tablet_replica.cc:333] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2: stopping tablet replica
I20251026 14:58:00.143121 21512 raft_consensus.cc:2243] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 4 LEADER]: Raft consensus shutting down.
I20251026 14:58:00.143854 21512 raft_consensus.cc:2272] T 2338d7ddd3a54537a9fbc8f7fc46de45 P 21eada14be174e129814165b15454da2 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.145534 21512 tablet_replica.cc:333] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2: stopping tablet replica
I20251026 14:58:00.145947 21512 raft_consensus.cc:2243] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 5 LEADER]: Raft consensus shutting down.
I20251026 14:58:00.146690 21512 raft_consensus.cc:2272] T 859bffc81a9b4a248c1b284e6bc3d41b P 21eada14be174e129814165b15454da2 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.148286 21512 tablet_replica.cc:333] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2: stopping tablet replica
I20251026 14:58:00.148692 21512 raft_consensus.cc:2243] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:00.149081 21512 raft_consensus.cc:2272] T d12e45731a5146be91de773fcc96acf4 P 21eada14be174e129814165b15454da2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.236938 21512 tablet_server.cc:195] TabletServer@127.21.2.1:0 shutdown complete.
I20251026 14:58:00.256575 21512 tablet_server.cc:178] TabletServer@127.21.2.2:0 shutting down...
I20251026 14:58:00.291944 21512 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 14:58:00.292690 21512 tablet_replica.cc:333] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5: stopping tablet replica
I20251026 14:58:00.293323 21512 raft_consensus.cc:2243] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 LEADER]: Raft consensus shutting down.
I20251026 14:58:00.294143 21512 raft_consensus.cc:2272] T 1ecb97a63a3c422eb2079ce004a1c7ec P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.295965 21512 tablet_replica.cc:333] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5: stopping tablet replica
I20251026 14:58:00.296445 21512 raft_consensus.cc:2243] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:00.296855 21512 raft_consensus.cc:2272] T d676fec08f2a45b8b44e1e1574606e6a P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.298662 21512 tablet_replica.cc:333] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5: stopping tablet replica
I20251026 14:58:00.299131 21512 raft_consensus.cc:2243] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:00.299500 21512 raft_consensus.cc:2272] T 2338d7ddd3a54537a9fbc8f7fc46de45 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.301196 21512 tablet_replica.cc:333] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5: stopping tablet replica
I20251026 14:58:00.301748 21512 raft_consensus.cc:2243] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:00.302124 21512 raft_consensus.cc:2272] T d4910c8c220b4256913e4ab10a75ebe1 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.303709 21512 tablet_replica.cc:333] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5: stopping tablet replica
I20251026 14:58:00.304138 21512 raft_consensus.cc:2243] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:00.304617 21512 raft_consensus.cc:2272] T fc1d64845a9543119c22b77b1256f9f3 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.306471 21512 tablet_replica.cc:333] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5: stopping tablet replica
I20251026 14:58:00.306934 21512 raft_consensus.cc:2243] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 LEADER]: Raft consensus shutting down.
I20251026 14:58:00.307757 21512 raft_consensus.cc:2272] T 0e31292ae87b4a04a394a31ca556654b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.309420 21512 tablet_replica.cc:333] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5: stopping tablet replica
I20251026 14:58:00.309875 21512 raft_consensus.cc:2243] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 4 LEADER]: Raft consensus shutting down.
I20251026 14:58:00.310662 21512 raft_consensus.cc:2272] T 90496135145648aba3eb19b0190c35c6 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.312229 21512 tablet_replica.cc:333] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5: stopping tablet replica
I20251026 14:58:00.312731 21512 raft_consensus.cc:2243] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:00.313138 21512 raft_consensus.cc:2272] T d12e45731a5146be91de773fcc96acf4 P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.314813 21512 tablet_replica.cc:333] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5: stopping tablet replica
I20251026 14:58:00.315248 21512 raft_consensus.cc:2243] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:00.315635 21512 raft_consensus.cc:2272] T 859bffc81a9b4a248c1b284e6bc3d41b P d2d4f51a45dc439ea67c2fe81ed6b1a5 [term 5 FOLLOWER]: Raft consensus is shut down!
W20251026 14:58:00.333716 21760 consensus_peers.cc:597] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57 -> Peer 21eada14be174e129814165b15454da2 (127.21.2.1:32983): Couldn't send request to peer 21eada14be174e129814165b15454da2. Status: Network error: Client connection negotiation failed: client connection to 127.21.2.1:32983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:00.343418 21512 tablet_server.cc:195] TabletServer@127.21.2.2:0 shutdown complete.
W20251026 14:58:00.347657 21760 consensus_peers.cc:597] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57 -> Peer 21eada14be174e129814165b15454da2 (127.21.2.1:32983): Couldn't send request to peer 21eada14be174e129814165b15454da2. Status: Network error: Client connection negotiation failed: client connection to 127.21.2.1:32983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:00.361639 21512 tablet_server.cc:178] TabletServer@127.21.2.3:0 shutting down...
I20251026 14:58:00.388497 21512 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 14:58:00.389184 21512 tablet_replica.cc:333] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57: stopping tablet replica
I20251026 14:58:00.389851 21512 raft_consensus.cc:2243] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57 [term 2 LEADER]: Raft consensus shutting down.
I20251026 14:58:00.390767 21512 raft_consensus.cc:2272] T d4910c8c220b4256913e4ab10a75ebe1 P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.392625 21512 tablet_replica.cc:333] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57: stopping tablet replica
I20251026 14:58:00.393129 21512 raft_consensus.cc:2243] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57 [term 2 LEADER]: Raft consensus shutting down.
I20251026 14:58:00.393968 21512 raft_consensus.cc:2272] T d676fec08f2a45b8b44e1e1574606e6a P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.395643 21512 tablet_replica.cc:333] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57: stopping tablet replica
I20251026 14:58:00.396140 21512 raft_consensus.cc:2243] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:00.396582 21512 raft_consensus.cc:2272] T 2338d7ddd3a54537a9fbc8f7fc46de45 P cdc74a031aba4e589df4c9eb8590cc57 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.398278 21512 tablet_replica.cc:333] T fc1d64845a9543119c22b77b1256f9f3 P cdc74a031aba4e589df4c9eb8590cc57: stopping tablet replica
I20251026 14:58:00.398770 21512 raft_consensus.cc:2243] T fc1d64845a9543119c22b77b1256f9f3 P cdc74a031aba4e589df4c9eb8590cc57 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:00.399184 21512 raft_consensus.cc:2272] T fc1d64845a9543119c22b77b1256f9f3 P cdc74a031aba4e589df4c9eb8590cc57 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.400758 21512 tablet_replica.cc:333] T 0e31292ae87b4a04a394a31ca556654b P cdc74a031aba4e589df4c9eb8590cc57: stopping tablet replica
I20251026 14:58:00.401218 21512 raft_consensus.cc:2243] T 0e31292ae87b4a04a394a31ca556654b P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:00.401679 21512 raft_consensus.cc:2272] T 0e31292ae87b4a04a394a31ca556654b P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.403270 21512 tablet_replica.cc:333] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57: stopping tablet replica
I20251026 14:58:00.403753 21512 raft_consensus.cc:2243] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:00.404153 21512 raft_consensus.cc:2272] T 90496135145648aba3eb19b0190c35c6 P cdc74a031aba4e589df4c9eb8590cc57 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.405939 21512 tablet_replica.cc:333] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57: stopping tablet replica
I20251026 14:58:00.406517 21512 raft_consensus.cc:2243] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:00.406981 21512 raft_consensus.cc:2272] T 1ecb97a63a3c422eb2079ce004a1c7ec P cdc74a031aba4e589df4c9eb8590cc57 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.408617 21512 tablet_replica.cc:333] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57: stopping tablet replica
I20251026 14:58:00.409050 21512 raft_consensus.cc:2243] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:00.409480 21512 raft_consensus.cc:2272] T 859bffc81a9b4a248c1b284e6bc3d41b P cdc74a031aba4e589df4c9eb8590cc57 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.411041 21512 tablet_replica.cc:333] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57: stopping tablet replica
I20251026 14:58:00.411469 21512 raft_consensus.cc:2243] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [term 3 LEADER]: Raft consensus shutting down.
I20251026 14:58:00.412206 21512 raft_consensus.cc:2272] T d12e45731a5146be91de773fcc96acf4 P cdc74a031aba4e589df4c9eb8590cc57 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.457067 21512 tablet_server.cc:195] TabletServer@127.21.2.3:0 shutdown complete.
I20251026 14:58:00.474797 21512 master.cc:561] Master@127.21.2.62:43649 shutting down...
I20251026 14:58:00.686780 21512 raft_consensus.cc:2243] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e [term 1 LEADER]: Raft consensus shutting down.
I20251026 14:58:00.687399 21512 raft_consensus.cc:2272] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:00.687736 21512 tablet_replica.cc:333] T 00000000000000000000000000000000 P fafcec9d4a5842d88f3d10b38b75565e: stopping tablet replica
I20251026 14:58:00.707770 21512 master.cc:583] Master@127.21.2.62:43649 shutdown complete.
21eada14be174e129814165b15454da2 4
cdc74a031aba4e589df4c9eb8590cc57 1
d2d4f51a45dc439ea67c2fe81ed6b1a5 4
21eada14be174e129814165b15454da2 3
cdc74a031aba4e589df4c9eb8590cc57 3
d2d4f51a45dc439ea67c2fe81ed6b1a5 3
21eada14be174e129814165b15454da2 0
cdc74a031aba4e589df4c9eb8590cc57 1
d2d4f51a45dc439ea67c2fe81ed6b1a5 8
21eada14be174e129814165b15454da2 3
cdc74a031aba4e589df4c9eb8590cc57 3
d2d4f51a45dc439ea67c2fe81ed6b1a5 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (31238 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20251026 14:58:00.748456 21512 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.21.2.62:39599
I20251026 14:58:00.749671 21512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 14:58:00.755211 22031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 14:58:00.756575 22034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 14:58:00.757328 22032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 14:58:00.758136 21512 server_base.cc:1047] running on GCE node
I20251026 14:58:00.759084 21512 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 14:58:00.759275 21512 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 14:58:00.759428 21512 hybrid_clock.cc:648] HybridClock initialized: now 1761490680759409 us; error 0 us; skew 500 ppm
I20251026 14:58:00.759968 21512 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 14:58:00.762396 21512 webserver.cc:492] Webserver started at http://127.21.2.62:35429/ using document root <none> and password file <none>
I20251026 14:58:00.762931 21512 fs_manager.cc:362] Metadata directory not provided
I20251026 14:58:00.763100 21512 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 14:58:00.763368 21512 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 14:58:00.764551 21512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761490649341556-21512-0/minicluster-data/master-0-root/instance:
uuid: "477cf6b76d7a46aa946907f6cc2c7024"
format_stamp: "Formatted at 2025-10-26 14:58:00 on dist-test-slave-9n4f"
I20251026 14:58:00.769181 21512 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20251026 14:58:00.772770 22039 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 14:58:00.773576 21512 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 14:58:00.773866 21512 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761490649341556-21512-0/minicluster-data/master-0-root
uuid: "477cf6b76d7a46aa946907f6cc2c7024"
format_stamp: "Formatted at 2025-10-26 14:58:00 on dist-test-slave-9n4f"
I20251026 14:58:00.774142 21512 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761490649341556-21512-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761490649341556-21512-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761490649341556-21512-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 14:58:00.788259 21512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 14:58:00.789580 21512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 14:58:00.829370 21512 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.62:39599
I20251026 14:58:00.829515 22090 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.2.62:39599 every 8 connection(s)
I20251026 14:58:00.833338 22091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:00.843143 22091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024: Bootstrap starting.
I20251026 14:58:00.847821 22091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:00.851989 22091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024: No bootstrap required, opened a new log
I20251026 14:58:00.854156 22091 raft_consensus.cc:359] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477cf6b76d7a46aa946907f6cc2c7024" member_type: VOTER }
I20251026 14:58:00.854597 22091 raft_consensus.cc:385] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:00.854869 22091 raft_consensus.cc:740] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 477cf6b76d7a46aa946907f6cc2c7024, State: Initialized, Role: FOLLOWER
I20251026 14:58:00.855448 22091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477cf6b76d7a46aa946907f6cc2c7024" member_type: VOTER }
I20251026 14:58:00.855911 22091 raft_consensus.cc:399] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 14:58:00.856127 22091 raft_consensus.cc:493] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 14:58:00.856392 22091 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:00.860412 22091 raft_consensus.cc:515] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477cf6b76d7a46aa946907f6cc2c7024" member_type: VOTER }
I20251026 14:58:00.860941 22091 leader_election.cc:304] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 477cf6b76d7a46aa946907f6cc2c7024; no voters:
I20251026 14:58:00.862169 22091 leader_election.cc:290] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 14:58:00.862581 22094 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:00.863915 22094 raft_consensus.cc:697] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024 [term 1 LEADER]: Becoming Leader. State: Replica: 477cf6b76d7a46aa946907f6cc2c7024, State: Running, Role: LEADER
I20251026 14:58:00.864634 22094 consensus_queue.cc:237] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477cf6b76d7a46aa946907f6cc2c7024" member_type: VOTER }
I20251026 14:58:00.865171 22091 sys_catalog.cc:565] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 14:58:00.869441 22095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "477cf6b76d7a46aa946907f6cc2c7024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477cf6b76d7a46aa946907f6cc2c7024" member_type: VOTER } }
I20251026 14:58:00.869643 22096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 477cf6b76d7a46aa946907f6cc2c7024. Latest consensus state: current_term: 1 leader_uuid: "477cf6b76d7a46aa946907f6cc2c7024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477cf6b76d7a46aa946907f6cc2c7024" member_type: VOTER } }
I20251026 14:58:00.870257 22095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024 [sys.catalog]: This master's current role is: LEADER
I20251026 14:58:00.870270 22096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024 [sys.catalog]: This master's current role is: LEADER
I20251026 14:58:00.873205 22101 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 14:58:00.878470 22101 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 14:58:00.882110 21512 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 14:58:00.888088 22101 catalog_manager.cc:1357] Generated new cluster ID: 7c9078e41e0b46d4af43c26a36716abd
I20251026 14:58:00.888391 22101 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 14:58:00.906644 22101 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 14:58:00.908042 22101 catalog_manager.cc:1514] Loading token signing keys...
I20251026 14:58:00.921461 22101 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024: Generated new TSK 0
I20251026 14:58:00.922087 22101 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 14:58:00.949313 21512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 14:58:00.955166 22112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 14:58:00.956308 22113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 14:58:00.959757 21512 server_base.cc:1047] running on GCE node
W20251026 14:58:00.959728 22115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 14:58:00.960911 21512 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 14:58:00.961094 21512 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 14:58:00.961215 21512 hybrid_clock.cc:648] HybridClock initialized: now 1761490680961202 us; error 0 us; skew 500 ppm
I20251026 14:58:00.961771 21512 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 14:58:00.964113 21512 webserver.cc:492] Webserver started at http://127.21.2.1:46359/ using document root <none> and password file <none>
I20251026 14:58:00.964608 21512 fs_manager.cc:362] Metadata directory not provided
I20251026 14:58:00.964773 21512 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 14:58:00.964993 21512 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 14:58:00.966161 21512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761490649341556-21512-0/minicluster-data/ts-0-root/instance:
uuid: "f6b6590ae6a440d1bcb913208797d2ea"
format_stamp: "Formatted at 2025-10-26 14:58:00 on dist-test-slave-9n4f"
I20251026 14:58:00.970674 21512 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20251026 14:58:00.974004 22120 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 14:58:00.974758 21512 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 14:58:00.975032 21512 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761490649341556-21512-0/minicluster-data/ts-0-root
uuid: "f6b6590ae6a440d1bcb913208797d2ea"
format_stamp: "Formatted at 2025-10-26 14:58:00 on dist-test-slave-9n4f"
I20251026 14:58:00.975333 21512 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761490649341556-21512-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761490649341556-21512-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761490649341556-21512-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 14:58:00.987795 21512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 14:58:00.989051 21512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 14:58:00.990578 21512 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 14:58:00.992925 21512 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 14:58:00.993134 21512 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 14:58:00.993391 21512 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 14:58:00.993562 21512 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 14:58:01.034194 21512 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.1:36141
I20251026 14:58:01.034189 22182 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.2.1:36141 every 8 connection(s)
I20251026 14:58:01.039778 21512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 14:58:01.047142 22187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 14:58:01.048650 22188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 14:58:01.051363 22190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 14:58:01.052064 21512 server_base.cc:1047] running on GCE node
I20251026 14:58:01.052021 22183 heartbeater.cc:344] Connected to a master server at 127.21.2.62:39599
I20251026 14:58:01.052644 22183 heartbeater.cc:461] Registering TS with master...
I20251026 14:58:01.053400 22183 heartbeater.cc:507] Master 127.21.2.62:39599 requested a full tablet report, sending...
I20251026 14:58:01.053570 21512 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 14:58:01.053824 21512 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 14:58:01.053978 21512 hybrid_clock.cc:648] HybridClock initialized: now 1761490681053962 us; error 0 us; skew 500 ppm
I20251026 14:58:01.054661 21512 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 14:58:01.055711 22056 ts_manager.cc:194] Registered new tserver with Master: f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141)
I20251026 14:58:01.057396 21512 webserver.cc:492] Webserver started at http://127.21.2.2:38271/ using document root <none> and password file <none>
I20251026 14:58:01.057762 22056 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48774
I20251026 14:58:01.057895 21512 fs_manager.cc:362] Metadata directory not provided
I20251026 14:58:01.058138 21512 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 14:58:01.058424 21512 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 14:58:01.059687 21512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761490649341556-21512-0/minicluster-data/ts-1-root/instance:
uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
format_stamp: "Formatted at 2025-10-26 14:58:01 on dist-test-slave-9n4f"
I20251026 14:58:01.064805 21512 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20251026 14:58:01.069149 22195 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 14:58:01.070211 21512 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251026 14:58:01.070555 21512 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761490649341556-21512-0/minicluster-data/ts-1-root
uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
format_stamp: "Formatted at 2025-10-26 14:58:01 on dist-test-slave-9n4f"
I20251026 14:58:01.070878 21512 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761490649341556-21512-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761490649341556-21512-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761490649341556-21512-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 14:58:01.100193 21512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 14:58:01.101830 21512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 14:58:01.103462 21512 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 14:58:01.105837 21512 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 14:58:01.106051 21512 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 14:58:01.106278 21512 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 14:58:01.106441 21512 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 14:58:01.147151 21512 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.2:38737
I20251026 14:58:01.147244 22257 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.2.2:38737 every 8 connection(s)
I20251026 14:58:01.151753 21512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 14:58:01.158442 22261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 14:58:01.159387 22262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 14:58:01.162189 22264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 14:58:01.162422 21512 server_base.cc:1047] running on GCE node
I20251026 14:58:01.163568 21512 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 14:58:01.163786 21512 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 14:58:01.163960 22258 heartbeater.cc:344] Connected to a master server at 127.21.2.62:39599
I20251026 14:58:01.163959 21512 hybrid_clock.cc:648] HybridClock initialized: now 1761490681163939 us; error 0 us; skew 500 ppm
I20251026 14:58:01.164502 22258 heartbeater.cc:461] Registering TS with master...
I20251026 14:58:01.164994 21512 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 14:58:01.165498 22258 heartbeater.cc:507] Master 127.21.2.62:39599 requested a full tablet report, sending...
I20251026 14:58:01.167547 22056 ts_manager.cc:194] Registered new tserver with Master: 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737)
I20251026 14:58:01.168109 21512 webserver.cc:492] Webserver started at http://127.21.2.3:43009/ using document root <none> and password file <none>
I20251026 14:58:01.168814 21512 fs_manager.cc:362] Metadata directory not provided
I20251026 14:58:01.169009 22056 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48778
I20251026 14:58:01.169031 21512 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 14:58:01.169648 21512 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 14:58:01.171337 21512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761490649341556-21512-0/minicluster-data/ts-2-root/instance:
uuid: "12bdf853a41f4c14a37d70908b8c2640"
format_stamp: "Formatted at 2025-10-26 14:58:01 on dist-test-slave-9n4f"
I20251026 14:58:01.176368 21512 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20251026 14:58:01.179975 22269 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 14:58:01.180770 21512 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251026 14:58:01.181071 21512 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761490649341556-21512-0/minicluster-data/ts-2-root
uuid: "12bdf853a41f4c14a37d70908b8c2640"
format_stamp: "Formatted at 2025-10-26 14:58:01 on dist-test-slave-9n4f"
I20251026 14:58:01.181432 21512 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761490649341556-21512-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761490649341556-21512-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1761490649341556-21512-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 14:58:01.197223 21512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 14:58:01.198468 21512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 14:58:01.199900 21512 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 14:58:01.202152 21512 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 14:58:01.202376 21512 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 14:58:01.202615 21512 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 14:58:01.202782 21512 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 14:58:01.241969 21512 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.3:40117
I20251026 14:58:01.242071 22331 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.2.3:40117 every 8 connection(s)
I20251026 14:58:01.255914 22332 heartbeater.cc:344] Connected to a master server at 127.21.2.62:39599
I20251026 14:58:01.256280 22332 heartbeater.cc:461] Registering TS with master...
I20251026 14:58:01.256995 22332 heartbeater.cc:507] Master 127.21.2.62:39599 requested a full tablet report, sending...
I20251026 14:58:01.258906 22056 ts_manager.cc:194] Registered new tserver with Master: 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:01.259012 21512 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013946209s
I20251026 14:58:01.260210 22056 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48784
I20251026 14:58:01.261384 21512 test_util.cc:276] Using random seed: 39120175
I20251026 14:58:01.284260 22056 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48796:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 14:58:01.287177 22056 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 14:58:01.361258 22148 tablet_service.cc:1505] Processing CreateTablet for tablet 304dea4666704f0c81d11c8ed58593cc (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION VALUES < 214748364
I20251026 14:58:01.363368 22148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 304dea4666704f0c81d11c8ed58593cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.366247 22147 tablet_service.cc:1505] Processing CreateTablet for tablet 638b4cb4614f49578bd04fc043b057f3 (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20251026 14:58:01.368247 22147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 638b4cb4614f49578bd04fc043b057f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.370961 22146 tablet_service.cc:1505] Processing CreateTablet for tablet 2cf56c1758324aebad5521cae8b061aa (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20251026 14:58:01.372939 22146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cf56c1758324aebad5521cae8b061aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.375283 22145 tablet_service.cc:1505] Processing CreateTablet for tablet da9afe4ffb9547c0b7ff3ed93b58248f (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20251026 14:58:01.377722 22145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da9afe4ffb9547c0b7ff3ed93b58248f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.379122 22297 tablet_service.cc:1505] Processing CreateTablet for tablet 304dea4666704f0c81d11c8ed58593cc (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION VALUES < 214748364
I20251026 14:58:01.380568 22297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 304dea4666704f0c81d11c8ed58593cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.383984 22296 tablet_service.cc:1505] Processing CreateTablet for tablet 638b4cb4614f49578bd04fc043b057f3 (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20251026 14:58:01.385435 22296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 638b4cb4614f49578bd04fc043b057f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.390770 22295 tablet_service.cc:1505] Processing CreateTablet for tablet 2cf56c1758324aebad5521cae8b061aa (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20251026 14:58:01.392642 22295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cf56c1758324aebad5521cae8b061aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.420202 22294 tablet_service.cc:1505] Processing CreateTablet for tablet da9afe4ffb9547c0b7ff3ed93b58248f (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20251026 14:58:01.422420 22294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da9afe4ffb9547c0b7ff3ed93b58248f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.429195 22293 tablet_service.cc:1505] Processing CreateTablet for tablet 4a8c1bc474434dd88d5a7eceeeece989 (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20251026 14:58:01.437798 22293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a8c1bc474434dd88d5a7eceeeece989. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.442512 22292 tablet_service.cc:1505] Processing CreateTablet for tablet 9e79d8a8b9394af5b39918f8a0a43b39 (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20251026 14:58:01.442790 22145 tablet_service.cc:1505] Processing CreateTablet for tablet 9e79d8a8b9394af5b39918f8a0a43b39 (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20251026 14:58:01.444607 22292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e79d8a8b9394af5b39918f8a0a43b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.444958 22145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e79d8a8b9394af5b39918f8a0a43b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.447875 22347 tablet_bootstrap.cc:492] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea: Bootstrap starting.
I20251026 14:58:01.446771 22148 tablet_service.cc:1505] Processing CreateTablet for tablet 4a8c1bc474434dd88d5a7eceeeece989 (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20251026 14:58:01.457887 22223 tablet_service.cc:1505] Processing CreateTablet for tablet 304dea4666704f0c81d11c8ed58593cc (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION VALUES < 214748364
I20251026 14:58:01.458143 22222 tablet_service.cc:1505] Processing CreateTablet for tablet 638b4cb4614f49578bd04fc043b057f3 (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20251026 14:58:01.459422 22223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 304dea4666704f0c81d11c8ed58593cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.459264 22220 tablet_service.cc:1505] Processing CreateTablet for tablet da9afe4ffb9547c0b7ff3ed93b58248f (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20251026 14:58:01.461895 22219 tablet_service.cc:1505] Processing CreateTablet for tablet 4a8c1bc474434dd88d5a7eceeeece989 (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20251026 14:58:01.463999 22221 tablet_service.cc:1505] Processing CreateTablet for tablet 2cf56c1758324aebad5521cae8b061aa (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20251026 14:58:01.466244 22218 tablet_service.cc:1505] Processing CreateTablet for tablet 9e79d8a8b9394af5b39918f8a0a43b39 (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20251026 14:58:01.458356 22148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a8c1bc474434dd88d5a7eceeeece989. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.476840 22220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da9afe4ffb9547c0b7ff3ed93b58248f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.480413 22347 tablet_bootstrap.cc:654] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:01.482218 22219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a8c1bc474434dd88d5a7eceeeece989. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.486299 22221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cf56c1758324aebad5521cae8b061aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.490789 22222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 638b4cb4614f49578bd04fc043b057f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.493686 22218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e79d8a8b9394af5b39918f8a0a43b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.494877 22291 tablet_service.cc:1505] Processing CreateTablet for tablet f4b102dcc6534e2583371c5b81e2da5b (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20251026 14:58:01.496855 22291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4b102dcc6534e2583371c5b81e2da5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.498119 22217 tablet_service.cc:1505] Processing CreateTablet for tablet f4b102dcc6534e2583371c5b81e2da5b (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20251026 14:58:01.500523 22217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4b102dcc6534e2583371c5b81e2da5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.510318 22146 tablet_service.cc:1505] Processing CreateTablet for tablet f4b102dcc6534e2583371c5b81e2da5b (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20251026 14:58:01.511813 22146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4b102dcc6534e2583371c5b81e2da5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.536087 22350 tablet_bootstrap.cc:492] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced: Bootstrap starting.
I20251026 14:58:01.544027 22350 tablet_bootstrap.cc:654] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:01.550130 22348 tablet_bootstrap.cc:492] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640: Bootstrap starting.
I20251026 14:58:01.557324 22218 tablet_service.cc:1505] Processing CreateTablet for tablet c7ad349a3a2c4e6e8235a080aab372b2 (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20251026 14:58:01.558738 22218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7ad349a3a2c4e6e8235a080aab372b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.561578 22145 tablet_service.cc:1505] Processing CreateTablet for tablet c7ad349a3a2c4e6e8235a080aab372b2 (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20251026 14:58:01.563316 22145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7ad349a3a2c4e6e8235a080aab372b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.565831 22347 tablet_bootstrap.cc:492] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea: No bootstrap required, opened a new log
I20251026 14:58:01.566627 22347 ts_tablet_manager.cc:1403] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea: Time spent bootstrapping tablet: real 0.119s user 0.022s sys 0.024s
I20251026 14:58:01.571332 22347 raft_consensus.cc:359] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.572130 22347 raft_consensus.cc:385] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:01.573873 22347 raft_consensus.cc:740] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Initialized, Role: FOLLOWER
I20251026 14:58:01.575956 22348 tablet_bootstrap.cc:654] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:01.576081 22347 consensus_queue.cc:260] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.577831 22350 tablet_bootstrap.cc:492] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced: No bootstrap required, opened a new log
I20251026 14:58:01.583168 22350 ts_tablet_manager.cc:1403] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced: Time spent bootstrapping tablet: real 0.047s user 0.015s sys 0.018s
I20251026 14:58:01.585188 22291 tablet_service.cc:1505] Processing CreateTablet for tablet c7ad349a3a2c4e6e8235a080aab372b2 (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20251026 14:58:01.586618 22291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7ad349a3a2c4e6e8235a080aab372b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.589051 22146 tablet_service.cc:1505] Processing CreateTablet for tablet 9e0f3fab0c2e40daa0e161d82949044d (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20251026 14:58:01.592041 22146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e0f3fab0c2e40daa0e161d82949044d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.590304 22222 tablet_service.cc:1505] Processing CreateTablet for tablet 9e0f3fab0c2e40daa0e161d82949044d (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20251026 14:58:01.594305 22350 raft_consensus.cc:359] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.595170 22350 raft_consensus.cc:385] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:01.595785 22350 raft_consensus.cc:740] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Initialized, Role: FOLLOWER
I20251026 14:58:01.596674 22350 consensus_queue.cc:260] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.597822 22294 tablet_service.cc:1505] Processing CreateTablet for tablet 9e0f3fab0c2e40daa0e161d82949044d (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20251026 14:58:01.600554 22148 tablet_service.cc:1505] Processing CreateTablet for tablet e55b543aef104e7290aa5dae37b421cb (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20251026 14:58:01.595204 22222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e0f3fab0c2e40daa0e161d82949044d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.602927 22347 ts_tablet_manager.cc:1434] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea: Time spent starting tablet: real 0.036s user 0.013s sys 0.007s
I20251026 14:58:01.602838 22183 heartbeater.cc:499] Master 127.21.2.62:39599 was elected leader, sending a full tablet report...
I20251026 14:58:01.604111 22347 tablet_bootstrap.cc:492] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea: Bootstrap starting.
I20251026 14:58:01.609356 22294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e0f3fab0c2e40daa0e161d82949044d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.617533 22148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e55b543aef104e7290aa5dae37b421cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.631487 22347 tablet_bootstrap.cc:654] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:01.634176 22354 raft_consensus.cc:493] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:01.631830 22348 tablet_bootstrap.cc:492] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640: No bootstrap required, opened a new log
I20251026 14:58:01.638792 22348 ts_tablet_manager.cc:1403] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640: Time spent bootstrapping tablet: real 0.089s user 0.011s sys 0.008s
I20251026 14:58:01.638496 22354 raft_consensus.cc:515] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.640352 22218 tablet_service.cc:1505] Processing CreateTablet for tablet e55b543aef104e7290aa5dae37b421cb (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20251026 14:58:01.642619 22348 raft_consensus.cc:359] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.643438 22348 raft_consensus.cc:385] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:01.643800 22348 raft_consensus.cc:740] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Initialized, Role: FOLLOWER
I20251026 14:58:01.644788 22348 consensus_queue.cc:260] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.646468 22218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e55b543aef104e7290aa5dae37b421cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.652911 22158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a8c1bc474434dd88d5a7eceeeece989" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea" is_pre_election: true
W20251026 14:58:01.654958 22199 leader_election.cc:343] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:01.656011 22332 heartbeater.cc:499] Master 127.21.2.62:39599 was elected leader, sending a full tablet report...
I20251026 14:58:01.635190 22350 ts_tablet_manager.cc:1434] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced: Time spent starting tablet: real 0.044s user 0.015s sys 0.005s
I20251026 14:58:01.635691 22258 heartbeater.cc:499] Master 127.21.2.62:39599 was elected leader, sending a full tablet report...
I20251026 14:58:01.658182 22350 tablet_bootstrap.cc:492] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced: Bootstrap starting.
I20251026 14:58:01.657171 22348 ts_tablet_manager.cc:1434] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640: Time spent starting tablet: real 0.018s user 0.005s sys 0.006s
I20251026 14:58:01.662784 22348 tablet_bootstrap.cc:492] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640: Bootstrap starting.
I20251026 14:58:01.665812 22350 tablet_bootstrap.cc:654] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:01.671594 22348 tablet_bootstrap.cc:654] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:01.685907 22354 leader_election.cc:290] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:01.689007 22348 tablet_bootstrap.cc:492] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640: No bootstrap required, opened a new log
I20251026 14:58:01.689684 22348 ts_tablet_manager.cc:1403] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.008s
I20251026 14:58:01.691175 22347 tablet_bootstrap.cc:492] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea: No bootstrap required, opened a new log
I20251026 14:58:01.691766 22347 ts_tablet_manager.cc:1403] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea: Time spent bootstrapping tablet: real 0.088s user 0.029s sys 0.011s
I20251026 14:58:01.694798 22347 raft_consensus.cc:359] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.695647 22347 raft_consensus.cc:385] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:01.695992 22347 raft_consensus.cc:740] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Initialized, Role: FOLLOWER
I20251026 14:58:01.697108 22307 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a8c1bc474434dd88d5a7eceeeece989" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bdf853a41f4c14a37d70908b8c2640" is_pre_election: true
I20251026 14:58:01.697741 22353 raft_consensus.cc:493] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:01.698243 22353 raft_consensus.cc:515] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
W20251026 14:58:01.698932 22198 leader_election.cc:343] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:01.699381 22198 leader_election.cc:304] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b40ffb1f6864b108c15f0b8aefeaced; no voters: 12bdf853a41f4c14a37d70908b8c2640, f6b6590ae6a440d1bcb913208797d2ea
I20251026 14:58:01.699281 22291 tablet_service.cc:1505] Processing CreateTablet for tablet e55b543aef104e7290aa5dae37b421cb (DEFAULT_TABLE table=test-workload [id=4ba162ae900140c5841fd2e75c388f64]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20251026 14:58:01.698181 22357 raft_consensus.cc:493] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:01.701435 22354 raft_consensus.cc:2749] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:01.701184 22357 raft_consensus.cc:515] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.702620 22353 leader_election.cc:290] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:01.710350 22357 leader_election.cc:290] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737)
I20251026 14:58:01.712337 22350 tablet_bootstrap.cc:492] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced: No bootstrap required, opened a new log
I20251026 14:58:01.719331 22350 ts_tablet_manager.cc:1403] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced: Time spent bootstrapping tablet: real 0.061s user 0.020s sys 0.013s
I20251026 14:58:01.720734 22158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "638b4cb4614f49578bd04fc043b057f3" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea" is_pre_election: true
I20251026 14:58:01.722476 22350 raft_consensus.cc:359] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.724857 22291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e55b543aef104e7290aa5dae37b421cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:01.700407 22347 consensus_queue.cc:260] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.729094 22158 raft_consensus.cc:2468] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 0.
I20251026 14:58:01.732851 22273 leader_election.cc:304] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:01.733985 22357 raft_consensus.cc:2804] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:01.734431 22357 raft_consensus.cc:493] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:01.734946 22357 raft_consensus.cc:3060] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:01.737764 22348 raft_consensus.cc:359] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.738544 22348 raft_consensus.cc:385] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:01.738893 22348 raft_consensus.cc:740] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Initialized, Role: FOLLOWER
I20251026 14:58:01.739725 22348 consensus_queue.cc:260] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.741158 22307 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da9afe4ffb9547c0b7ff3ed93b58248f" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bdf853a41f4c14a37d70908b8c2640" is_pre_election: true
W20251026 14:58:01.742936 22123 leader_election.cc:343] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:01.744278 22347 ts_tablet_manager.cc:1434] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea: Time spent starting tablet: real 0.052s user 0.005s sys 0.000s
I20251026 14:58:01.723330 22350 raft_consensus.cc:385] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:01.746100 22350 raft_consensus.cc:740] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Initialized, Role: FOLLOWER
I20251026 14:58:01.746843 22350 consensus_queue.cc:260] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.752089 22357 raft_consensus.cc:515] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.756449 22347 tablet_bootstrap.cc:492] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea: Bootstrap starting.
I20251026 14:58:01.757040 22158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "638b4cb4614f49578bd04fc043b057f3" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
I20251026 14:58:01.757803 22158 raft_consensus.cc:3060] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:01.758765 22233 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "638b4cb4614f49578bd04fc043b057f3" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" is_pre_election: true
W20251026 14:58:01.760658 22271 leader_election.cc:343] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:01.761624 22350 ts_tablet_manager.cc:1434] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced: Time spent starting tablet: real 0.030s user 0.004s sys 0.000s
I20251026 14:58:01.762452 22357 leader_election.cc:290] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 election: Requested vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737)
I20251026 14:58:01.762933 22350 tablet_bootstrap.cc:492] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced: Bootstrap starting.
I20251026 14:58:01.764904 22233 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "638b4cb4614f49578bd04fc043b057f3" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
W20251026 14:58:01.766808 22271 leader_election.cc:343] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:01.767527 22357 raft_consensus.cc:493] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:01.768129 22357 raft_consensus.cc:515] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.771049 22233 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da9afe4ffb9547c0b7ff3ed93b58248f" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" is_pre_election: true
I20251026 14:58:01.771754 22158 raft_consensus.cc:2468] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 1.
I20251026 14:58:01.771888 22157 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e79d8a8b9394af5b39918f8a0a43b39" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea" is_pre_election: true
W20251026 14:58:01.772747 22122 leader_election.cc:343] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:01.772965 22273 leader_election.cc:304] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, f6b6590ae6a440d1bcb913208797d2ea; no voters: 8b40ffb1f6864b108c15f0b8aefeaced
I20251026 14:58:01.773238 22122 leader_election.cc:304] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6b6590ae6a440d1bcb913208797d2ea; no voters: 12bdf853a41f4c14a37d70908b8c2640, 8b40ffb1f6864b108c15f0b8aefeaced
W20251026 14:58:01.774268 22273 leader_election.cc:343] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:01.774852 22353 raft_consensus.cc:2749] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:01.774896 22233 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e79d8a8b9394af5b39918f8a0a43b39" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" is_pre_election: true
I20251026 14:58:01.775588 22357 leader_election.cc:290] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737)
I20251026 14:58:01.775750 22348 ts_tablet_manager.cc:1434] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640: Time spent starting tablet: real 0.084s user 0.007s sys 0.000s
I20251026 14:58:01.776127 22357 raft_consensus.cc:2804] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:01.776573 22357 raft_consensus.cc:697] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Becoming Leader. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Running, Role: LEADER
I20251026 14:58:01.777457 22357 consensus_queue.cc:237] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
W20251026 14:58:01.779428 22271 leader_election.cc:343] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:01.779945 22271 leader_election.cc:304] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640; no voters: 8b40ffb1f6864b108c15f0b8aefeaced, f6b6590ae6a440d1bcb913208797d2ea
I20251026 14:58:01.781224 22364 raft_consensus.cc:2749] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:01.781931 22348 tablet_bootstrap.cc:492] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640: Bootstrap starting.
I20251026 14:58:01.786828 22347 tablet_bootstrap.cc:654] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:01.790380 22348 tablet_bootstrap.cc:654] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:01.792800 22350 tablet_bootstrap.cc:654] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:01.799948 22055 catalog_manager.cc:5649] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "12bdf853a41f4c14a37d70908b8c2640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:01.803236 22348 tablet_bootstrap.cc:492] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640: No bootstrap required, opened a new log
I20251026 14:58:01.803761 22348 ts_tablet_manager.cc:1403] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.007s
I20251026 14:58:01.806524 22348 raft_consensus.cc:359] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.807224 22348 raft_consensus.cc:385] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:01.807523 22348 raft_consensus.cc:740] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Initialized, Role: FOLLOWER
I20251026 14:58:01.808221 22348 consensus_queue.cc:260] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.811981 22348 ts_tablet_manager.cc:1434] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20251026 14:58:01.812387 22350 tablet_bootstrap.cc:492] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced: No bootstrap required, opened a new log
I20251026 14:58:01.812572 22347 tablet_bootstrap.cc:492] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea: No bootstrap required, opened a new log
I20251026 14:58:01.812932 22350 ts_tablet_manager.cc:1403] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.014s
I20251026 14:58:01.813086 22347 ts_tablet_manager.cc:1403] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea: Time spent bootstrapping tablet: real 0.068s user 0.021s sys 0.010s
I20251026 14:58:01.813107 22348 tablet_bootstrap.cc:492] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640: Bootstrap starting.
I20251026 14:58:01.816030 22350 raft_consensus.cc:359] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.816320 22347 raft_consensus.cc:359] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.816958 22350 raft_consensus.cc:385] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:01.817235 22347 raft_consensus.cc:385] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:01.817399 22350 raft_consensus.cc:740] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Initialized, Role: FOLLOWER
I20251026 14:58:01.817750 22347 raft_consensus.cc:740] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Initialized, Role: FOLLOWER
I20251026 14:58:01.818437 22350 consensus_queue.cc:260] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.818691 22347 consensus_queue.cc:260] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.821027 22350 ts_tablet_manager.cc:1434] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20251026 14:58:01.821417 22347 ts_tablet_manager.cc:1434] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20251026 14:58:01.822240 22350 tablet_bootstrap.cc:492] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced: Bootstrap starting.
I20251026 14:58:01.822852 22348 tablet_bootstrap.cc:654] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:01.826736 22354 raft_consensus.cc:493] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:01.827280 22354 raft_consensus.cc:515] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.828785 22348 tablet_bootstrap.cc:492] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640: No bootstrap required, opened a new log
I20251026 14:58:01.829319 22348 ts_tablet_manager.cc:1403] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.001s
I20251026 14:58:01.829375 22354 leader_election.cc:290] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:01.830139 22350 tablet_bootstrap.cc:654] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:01.831580 22158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4b102dcc6534e2583371c5b81e2da5b" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea" is_pre_election: true
I20251026 14:58:01.832124 22347 tablet_bootstrap.cc:492] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea: Bootstrap starting.
I20251026 14:58:01.832312 22348 raft_consensus.cc:359] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.832784 22307 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4b102dcc6534e2583371c5b81e2da5b" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bdf853a41f4c14a37d70908b8c2640" is_pre_election: true
I20251026 14:58:01.833071 22348 raft_consensus.cc:385] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:01.833482 22348 raft_consensus.cc:740] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Initialized, Role: FOLLOWER
W20251026 14:58:01.834173 22199 leader_election.cc:343] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141): Illegal state: must be running to vote when last-logged opid is not known
W20251026 14:58:01.835253 22198 leader_election.cc:343] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:01.834990 22348 consensus_queue.cc:260] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.835778 22198 leader_election.cc:304] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b40ffb1f6864b108c15f0b8aefeaced; no voters: 12bdf853a41f4c14a37d70908b8c2640, f6b6590ae6a440d1bcb913208797d2ea
I20251026 14:58:01.838150 22348 ts_tablet_manager.cc:1434] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20251026 14:58:01.838925 22347 tablet_bootstrap.cc:654] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:01.839466 22348 tablet_bootstrap.cc:492] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640: Bootstrap starting.
I20251026 14:58:01.848189 22354 raft_consensus.cc:2749] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:01.848958 22348 tablet_bootstrap.cc:654] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:01.858165 22347 tablet_bootstrap.cc:492] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea: No bootstrap required, opened a new log
I20251026 14:58:01.858244 22350 tablet_bootstrap.cc:492] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced: No bootstrap required, opened a new log
I20251026 14:58:01.858204 22348 tablet_bootstrap.cc:492] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640: No bootstrap required, opened a new log
I20251026 14:58:01.858846 22347 ts_tablet_manager.cc:1403] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea: Time spent bootstrapping tablet: real 0.027s user 0.015s sys 0.000s
I20251026 14:58:01.858846 22350 ts_tablet_manager.cc:1403] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced: Time spent bootstrapping tablet: real 0.037s user 0.021s sys 0.010s
I20251026 14:58:01.858888 22348 ts_tablet_manager.cc:1403] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.008s
I20251026 14:58:01.861383 22348 raft_consensus.cc:359] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.861937 22348 raft_consensus.cc:385] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:01.861836 22347 raft_consensus.cc:359] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.862385 22348 raft_consensus.cc:740] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Initialized, Role: FOLLOWER
I20251026 14:58:01.862046 22350 raft_consensus.cc:359] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.862697 22347 raft_consensus.cc:385] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:01.862886 22350 raft_consensus.cc:385] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:01.863056 22347 raft_consensus.cc:740] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Initialized, Role: FOLLOWER
I20251026 14:58:01.863263 22350 raft_consensus.cc:740] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Initialized, Role: FOLLOWER
I20251026 14:58:01.863193 22348 consensus_queue.cc:260] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.863787 22347 consensus_queue.cc:260] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.864215 22350 consensus_queue.cc:260] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.865875 22348 ts_tablet_manager.cc:1434] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20251026 14:58:01.866545 22347 ts_tablet_manager.cc:1434] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20251026 14:58:01.866545 22350 ts_tablet_manager.cc:1434] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20251026 14:58:01.867059 22348 tablet_bootstrap.cc:492] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640: Bootstrap starting.
I20251026 14:58:01.871408 22347 tablet_bootstrap.cc:492] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea: Bootstrap starting.
I20251026 14:58:01.872066 22350 tablet_bootstrap.cc:492] T 638b4cb4614f49578bd04fc043b057f3 P 8b40ffb1f6864b108c15f0b8aefeaced: Bootstrap starting.
I20251026 14:58:01.874228 22348 tablet_bootstrap.cc:654] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640: Neither blocks nor log segments found. Creating new log.
W20251026 14:58:01.877079 22104 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251026 14:58:01.879061 22350 tablet_bootstrap.cc:654] T 638b4cb4614f49578bd04fc043b057f3 P 8b40ffb1f6864b108c15f0b8aefeaced: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:01.879061 22347 tablet_bootstrap.cc:654] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:01.893821 22348 tablet_bootstrap.cc:492] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640: No bootstrap required, opened a new log
I20251026 14:58:01.894325 22348 ts_tablet_manager.cc:1403] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.007s
I20251026 14:58:01.898424 22348 raft_consensus.cc:359] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.898990 22348 raft_consensus.cc:385] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:01.899238 22348 raft_consensus.cc:740] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Initialized, Role: FOLLOWER
I20251026 14:58:01.899844 22348 consensus_queue.cc:260] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.902012 22348 ts_tablet_manager.cc:1434] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251026 14:58:01.903003 22348 tablet_bootstrap.cc:492] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640: Bootstrap starting.
I20251026 14:58:01.907996 22347 tablet_bootstrap.cc:492] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea: No bootstrap required, opened a new log
I20251026 14:58:01.908535 22347 ts_tablet_manager.cc:1403] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.014s
I20251026 14:58:01.909058 22350 tablet_bootstrap.cc:492] T 638b4cb4614f49578bd04fc043b057f3 P 8b40ffb1f6864b108c15f0b8aefeaced: No bootstrap required, opened a new log
I20251026 14:58:01.909694 22350 ts_tablet_manager.cc:1403] T 638b4cb4614f49578bd04fc043b057f3 P 8b40ffb1f6864b108c15f0b8aefeaced: Time spent bootstrapping tablet: real 0.038s user 0.015s sys 0.017s
I20251026 14:58:01.911963 22348 tablet_bootstrap.cc:654] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:01.911629 22347 raft_consensus.cc:359] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.912405 22347 raft_consensus.cc:385] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:01.912734 22347 raft_consensus.cc:740] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Initialized, Role: FOLLOWER
I20251026 14:58:01.912519 22350 raft_consensus.cc:359] T 638b4cb4614f49578bd04fc043b057f3 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.913231 22350 raft_consensus.cc:385] T 638b4cb4614f49578bd04fc043b057f3 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:01.913563 22350 raft_consensus.cc:740] T 638b4cb4614f49578bd04fc043b057f3 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Initialized, Role: FOLLOWER
I20251026 14:58:01.913561 22347 consensus_queue.cc:260] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.914469 22350 consensus_queue.cc:260] T 638b4cb4614f49578bd04fc043b057f3 P 8b40ffb1f6864b108c15f0b8aefeaced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.916847 22350 ts_tablet_manager.cc:1434] T 638b4cb4614f49578bd04fc043b057f3 P 8b40ffb1f6864b108c15f0b8aefeaced: Time spent starting tablet: real 0.007s user 0.000s sys 0.004s
I20251026 14:58:01.917793 22350 tablet_bootstrap.cc:492] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced: Bootstrap starting.
I20251026 14:58:01.922108 22347 ts_tablet_manager.cc:1434] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea: Time spent starting tablet: real 0.013s user 0.008s sys 0.001s
I20251026 14:58:01.923177 22347 tablet_bootstrap.cc:492] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea: Bootstrap starting.
I20251026 14:58:01.923713 22348 tablet_bootstrap.cc:492] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640: No bootstrap required, opened a new log
I20251026 14:58:01.924229 22348 ts_tablet_manager.cc:1403] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.007s
I20251026 14:58:01.924299 22350 tablet_bootstrap.cc:654] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:01.927251 22348 raft_consensus.cc:359] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.928007 22348 raft_consensus.cc:385] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:01.928287 22348 raft_consensus.cc:740] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Initialized, Role: FOLLOWER
I20251026 14:58:01.929169 22348 consensus_queue.cc:260] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.930989 22347 tablet_bootstrap.cc:654] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:01.934134 22348 ts_tablet_manager.cc:1434] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20251026 14:58:01.935179 22348 tablet_bootstrap.cc:492] T 9e0f3fab0c2e40daa0e161d82949044d P 12bdf853a41f4c14a37d70908b8c2640: Bootstrap starting.
I20251026 14:58:01.941588 22348 tablet_bootstrap.cc:654] T 9e0f3fab0c2e40daa0e161d82949044d P 12bdf853a41f4c14a37d70908b8c2640: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:01.947450 22350 tablet_bootstrap.cc:492] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced: No bootstrap required, opened a new log
I20251026 14:58:01.947975 22350 ts_tablet_manager.cc:1403] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.014s
I20251026 14:58:01.950865 22350 raft_consensus.cc:359] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.951478 22350 raft_consensus.cc:385] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:01.951733 22350 raft_consensus.cc:740] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Initialized, Role: FOLLOWER
I20251026 14:58:01.952416 22347 tablet_bootstrap.cc:492] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea: No bootstrap required, opened a new log
I20251026 14:58:01.952350 22350 consensus_queue.cc:260] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.953112 22347 ts_tablet_manager.cc:1403] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.008s
I20251026 14:58:01.955488 22350 ts_tablet_manager.cc:1434] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20251026 14:58:01.956737 22350 tablet_bootstrap.cc:492] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced: Bootstrap starting.
I20251026 14:58:01.957726 22347 raft_consensus.cc:359] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.958770 22347 raft_consensus.cc:385] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:01.959172 22347 raft_consensus.cc:740] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Initialized, Role: FOLLOWER
I20251026 14:58:01.960299 22347 consensus_queue.cc:260] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.963569 22350 tablet_bootstrap.cc:654] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:01.963658 22348 tablet_bootstrap.cc:492] T 9e0f3fab0c2e40daa0e161d82949044d P 12bdf853a41f4c14a37d70908b8c2640: No bootstrap required, opened a new log
I20251026 14:58:01.964214 22348 ts_tablet_manager.cc:1403] T 9e0f3fab0c2e40daa0e161d82949044d P 12bdf853a41f4c14a37d70908b8c2640: Time spent bootstrapping tablet: real 0.029s user 0.015s sys 0.012s
I20251026 14:58:01.964651 22347 ts_tablet_manager.cc:1434] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20251026 14:58:01.965991 22347 tablet_bootstrap.cc:492] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea: Bootstrap starting.
I20251026 14:58:01.966358 22348 raft_consensus.cc:359] T 9e0f3fab0c2e40daa0e161d82949044d P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.967039 22348 raft_consensus.cc:385] T 9e0f3fab0c2e40daa0e161d82949044d P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:01.967308 22348 raft_consensus.cc:740] T 9e0f3fab0c2e40daa0e161d82949044d P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Initialized, Role: FOLLOWER
I20251026 14:58:01.967957 22348 consensus_queue.cc:260] T 9e0f3fab0c2e40daa0e161d82949044d P 12bdf853a41f4c14a37d70908b8c2640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:01.970136 22348 ts_tablet_manager.cc:1434] T 9e0f3fab0c2e40daa0e161d82949044d P 12bdf853a41f4c14a37d70908b8c2640: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251026 14:58:01.972158 22348 tablet_bootstrap.cc:492] T c7ad349a3a2c4e6e8235a080aab372b2 P 12bdf853a41f4c14a37d70908b8c2640: Bootstrap starting.
I20251026 14:58:01.973749 22347 tablet_bootstrap.cc:654] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:01.978531 22348 tablet_bootstrap.cc:654] T c7ad349a3a2c4e6e8235a080aab372b2 P 12bdf853a41f4c14a37d70908b8c2640: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:01.988190 22347 tablet_bootstrap.cc:492] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea: No bootstrap required, opened a new log
I20251026 14:58:01.989202 22347 ts_tablet_manager.cc:1403] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.012s
I20251026 14:58:02.002799 22350 tablet_bootstrap.cc:492] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced: No bootstrap required, opened a new log
I20251026 14:58:02.003243 22350 ts_tablet_manager.cc:1403] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced: Time spent bootstrapping tablet: real 0.047s user 0.020s sys 0.016s
I20251026 14:58:02.003878 22347 raft_consensus.cc:359] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.004956 22347 raft_consensus.cc:385] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:02.005357 22347 raft_consensus.cc:740] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Initialized, Role: FOLLOWER
I20251026 14:58:02.005538 22350 raft_consensus.cc:359] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.006148 22350 raft_consensus.cc:385] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:02.006428 22350 raft_consensus.cc:740] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Initialized, Role: FOLLOWER
I20251026 14:58:02.007191 22350 consensus_queue.cc:260] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.006366 22347 consensus_queue.cc:260] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.009083 22348 tablet_bootstrap.cc:492] T c7ad349a3a2c4e6e8235a080aab372b2 P 12bdf853a41f4c14a37d70908b8c2640: No bootstrap required, opened a new log
I20251026 14:58:02.009490 22350 ts_tablet_manager.cc:1434] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20251026 14:58:02.009604 22348 ts_tablet_manager.cc:1403] T c7ad349a3a2c4e6e8235a080aab372b2 P 12bdf853a41f4c14a37d70908b8c2640: Time spent bootstrapping tablet: real 0.038s user 0.025s sys 0.008s
I20251026 14:58:02.010563 22350 tablet_bootstrap.cc:492] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced: Bootstrap starting.
I20251026 14:58:02.010979 22347 ts_tablet_manager.cc:1434] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea: Time spent starting tablet: real 0.021s user 0.006s sys 0.001s
I20251026 14:58:02.011930 22347 tablet_bootstrap.cc:492] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea: Bootstrap starting.
I20251026 14:58:02.012228 22348 raft_consensus.cc:359] T c7ad349a3a2c4e6e8235a080aab372b2 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.012900 22348 raft_consensus.cc:385] T c7ad349a3a2c4e6e8235a080aab372b2 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:02.013154 22348 raft_consensus.cc:740] T c7ad349a3a2c4e6e8235a080aab372b2 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Initialized, Role: FOLLOWER
I20251026 14:58:02.013892 22348 consensus_queue.cc:260] T c7ad349a3a2c4e6e8235a080aab372b2 P 12bdf853a41f4c14a37d70908b8c2640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.016003 22348 ts_tablet_manager.cc:1434] T c7ad349a3a2c4e6e8235a080aab372b2 P 12bdf853a41f4c14a37d70908b8c2640: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251026 14:58:02.016950 22348 tablet_bootstrap.cc:492] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640: Bootstrap starting.
I20251026 14:58:02.017441 22350 tablet_bootstrap.cc:654] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:02.021248 22347 tablet_bootstrap.cc:654] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:02.022306 22348 tablet_bootstrap.cc:654] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:02.033900 22347 tablet_bootstrap.cc:492] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea: No bootstrap required, opened a new log
I20251026 14:58:02.034401 22347 ts_tablet_manager.cc:1403] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.012s
I20251026 14:58:02.036171 22347 raft_consensus.cc:359] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.036672 22347 raft_consensus.cc:385] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:02.036896 22347 raft_consensus.cc:740] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Initialized, Role: FOLLOWER
I20251026 14:58:02.039848 22348 tablet_bootstrap.cc:492] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640: No bootstrap required, opened a new log
I20251026 14:58:02.040347 22348 ts_tablet_manager.cc:1403] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.011s
I20251026 14:58:02.041887 22350 tablet_bootstrap.cc:492] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced: No bootstrap required, opened a new log
I20251026 14:58:02.042333 22350 ts_tablet_manager.cc:1403] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.013s
I20251026 14:58:02.039415 22347 consensus_queue.cc:260] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.042927 22348 raft_consensus.cc:359] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.043574 22348 raft_consensus.cc:385] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:02.043823 22348 raft_consensus.cc:740] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Initialized, Role: FOLLOWER
I20251026 14:58:02.044442 22347 ts_tablet_manager.cc:1434] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20251026 14:58:02.044921 22350 raft_consensus.cc:359] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.045509 22347 tablet_bootstrap.cc:492] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea: Bootstrap starting.
I20251026 14:58:02.045630 22350 raft_consensus.cc:385] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:02.044450 22348 consensus_queue.cc:260] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.047874 22348 ts_tablet_manager.cc:1434] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20251026 14:58:02.046034 22350 raft_consensus.cc:740] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Initialized, Role: FOLLOWER
I20251026 14:58:02.048712 22353 raft_consensus.cc:493] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:02.048960 22350 consensus_queue.cc:260] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.049218 22353 raft_consensus.cc:515] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.051213 22350 ts_tablet_manager.cc:1434] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20251026 14:58:02.051306 22353 leader_election.cc:290] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:02.052327 22347 tablet_bootstrap.cc:654] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:02.052922 22233 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7ad349a3a2c4e6e8235a080aab372b2" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" is_pre_election: true
I20251026 14:58:02.053412 22354 raft_consensus.cc:493] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:02.053683 22233 raft_consensus.cc:2468] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 0.
I20251026 14:58:02.053885 22354 raft_consensus.cc:515] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.055004 22122 leader_election.cc:304] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b40ffb1f6864b108c15f0b8aefeaced, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:02.055878 22354 leader_election.cc:290] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:02.057716 22307 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "304dea4666704f0c81d11c8ed58593cc" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bdf853a41f4c14a37d70908b8c2640" is_pre_election: true
I20251026 14:58:02.058341 22350 tablet_bootstrap.cc:492] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced: Bootstrap starting.
I20251026 14:58:02.058596 22307 raft_consensus.cc:2468] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b40ffb1f6864b108c15f0b8aefeaced in term 0.
I20251026 14:58:02.059640 22158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "304dea4666704f0c81d11c8ed58593cc" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea" is_pre_election: true
I20251026 14:58:02.060266 22158 raft_consensus.cc:2468] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b40ffb1f6864b108c15f0b8aefeaced in term 0.
I20251026 14:58:02.061316 22307 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7ad349a3a2c4e6e8235a080aab372b2" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bdf853a41f4c14a37d70908b8c2640" is_pre_election: true
I20251026 14:58:02.062220 22307 raft_consensus.cc:2468] T c7ad349a3a2c4e6e8235a080aab372b2 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 0.
I20251026 14:58:02.061467 22199 leader_election.cc:304] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b40ffb1f6864b108c15f0b8aefeaced, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:02.064772 22357 raft_consensus.cc:493] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:02.065297 22357 raft_consensus.cc:515] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.065627 22353 raft_consensus.cc:2804] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:02.066177 22353 raft_consensus.cc:493] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:02.066637 22353 raft_consensus.cc:3060] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.067430 22357 leader_election.cc:290] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737)
I20251026 14:58:02.069412 22158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da9afe4ffb9547c0b7ff3ed93b58248f" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea" is_pre_election: true
I20251026 14:58:02.069869 22354 raft_consensus.cc:2804] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:02.070071 22158 raft_consensus.cc:2468] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 0.
I20251026 14:58:02.070425 22354 raft_consensus.cc:493] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:02.070921 22354 raft_consensus.cc:3060] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.071350 22273 leader_election.cc:304] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:02.072870 22233 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da9afe4ffb9547c0b7ff3ed93b58248f" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" is_pre_election: true
I20251026 14:58:02.073531 22233 raft_consensus.cc:2468] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 0.
I20251026 14:58:02.075145 22357 raft_consensus.cc:2804] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:02.075568 22357 raft_consensus.cc:493] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:02.075973 22357 raft_consensus.cc:3060] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.083441 22357 raft_consensus.cc:515] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.083776 22353 raft_consensus.cc:515] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.085867 22354 raft_consensus.cc:515] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.086437 22158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da9afe4ffb9547c0b7ff3ed93b58248f" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
I20251026 14:58:02.086984 22158 raft_consensus.cc:3060] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.087466 22233 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7ad349a3a2c4e6e8235a080aab372b2" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
I20251026 14:58:02.088238 22233 raft_consensus.cc:3060] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.088589 22354 leader_election.cc:290] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 1 election: Requested vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:02.090193 22307 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "304dea4666704f0c81d11c8ed58593cc" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
I20251026 14:58:02.066637 22350 tablet_bootstrap.cc:654] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:02.090768 22307 raft_consensus.cc:3060] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.091318 22157 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "304dea4666704f0c81d11c8ed58593cc" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
I20251026 14:58:02.091902 22157 raft_consensus.cc:3060] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.094319 22357 leader_election.cc:290] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 election: Requested vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737)
I20251026 14:58:02.095826 22232 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da9afe4ffb9547c0b7ff3ed93b58248f" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
I20251026 14:58:02.096664 22232 raft_consensus.cc:3060] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.096699 22233 raft_consensus.cc:2468] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 1.
I20251026 14:58:02.097991 22307 raft_consensus.cc:2468] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b40ffb1f6864b108c15f0b8aefeaced in term 1.
I20251026 14:58:02.098152 22122 leader_election.cc:304] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b40ffb1f6864b108c15f0b8aefeaced, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:02.099179 22198 leader_election.cc:304] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, 8b40ffb1f6864b108c15f0b8aefeaced; no voters:
I20251026 14:58:02.100333 22354 raft_consensus.cc:2804] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:02.100956 22354 raft_consensus.cc:697] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 LEADER]: Becoming Leader. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Running, Role: LEADER
I20251026 14:58:02.101858 22354 consensus_queue.cc:237] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.103849 22357 raft_consensus.cc:493] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:02.104431 22357 raft_consensus.cc:515] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.105952 22306 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7ad349a3a2c4e6e8235a080aab372b2" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
I20251026 14:58:02.106567 22306 raft_consensus.cc:3060] T c7ad349a3a2c4e6e8235a080aab372b2 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.107405 22347 tablet_bootstrap.cc:492] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea: No bootstrap required, opened a new log
I20251026 14:58:02.107883 22347 ts_tablet_manager.cc:1403] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea: Time spent bootstrapping tablet: real 0.063s user 0.021s sys 0.022s
I20251026 14:58:02.108078 22156 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2cf56c1758324aebad5521cae8b061aa" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea" is_pre_election: true
I20251026 14:58:02.109019 22156 raft_consensus.cc:2468] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 0.
I20251026 14:58:02.110596 22347 raft_consensus.cc:359] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.111318 22347 raft_consensus.cc:385] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:02.111624 22347 raft_consensus.cc:740] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Initialized, Role: FOLLOWER
I20251026 14:58:02.110627 22231 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2cf56c1758324aebad5521cae8b061aa" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" is_pre_election: true
I20251026 14:58:02.112552 22231 raft_consensus.cc:2468] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 0.
I20251026 14:58:02.114162 22364 raft_consensus.cc:493] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:02.114939 22388 raft_consensus.cc:2804] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:02.115144 22306 raft_consensus.cc:2468] T c7ad349a3a2c4e6e8235a080aab372b2 P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 1.
I20251026 14:58:02.114961 22364 raft_consensus.cc:515] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.106515 22357 leader_election.cc:290] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737)
I20251026 14:58:02.118587 22271 leader_election.cc:304] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, 8b40ffb1f6864b108c15f0b8aefeaced; no voters:
I20251026 14:58:02.119963 22388 raft_consensus.cc:697] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 1 LEADER]: Becoming Leader. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Running, Role: LEADER
I20251026 14:58:02.120872 22388 consensus_queue.cc:237] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.122426 22156 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e55b543aef104e7290aa5dae37b421cb" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea" is_pre_election: true
I20251026 14:58:02.112428 22347 consensus_queue.cc:260] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
W20251026 14:58:02.124476 22273 leader_election.cc:343] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:02.125453 22364 leader_election.cc:290] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737)
I20251026 14:58:02.128443 22157 raft_consensus.cc:2468] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b40ffb1f6864b108c15f0b8aefeaced in term 1.
I20251026 14:58:02.127401 22158 raft_consensus.cc:2468] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 1.
I20251026 14:58:02.127543 22231 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e55b543aef104e7290aa5dae37b421cb" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" is_pre_election: true
I20251026 14:58:02.128479 22364 raft_consensus.cc:2804] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:02.125936 22347 ts_tablet_manager.cc:1434] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea: Time spent starting tablet: real 0.018s user 0.007s sys 0.001s
I20251026 14:58:02.128981 22353 leader_election.cc:290] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 1 election: Requested vote from peers 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:02.125064 22232 raft_consensus.cc:2468] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 1.
I20251026 14:58:02.146559 22054 catalog_manager.cc:5649] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1). New cstate: current_term: 1 leader_uuid: "f6b6590ae6a440d1bcb913208797d2ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: UNKNOWN } } }
W20251026 14:58:02.152820 22271 leader_election.cc:343] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:02.155838 22271 leader_election.cc:304] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640; no voters: 8b40ffb1f6864b108c15f0b8aefeaced, f6b6590ae6a440d1bcb913208797d2ea
I20251026 14:58:02.156699 22357 raft_consensus.cc:2749] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:02.153479 22347 tablet_bootstrap.cc:492] T e55b543aef104e7290aa5dae37b421cb P f6b6590ae6a440d1bcb913208797d2ea: Bootstrap starting.
I20251026 14:58:02.154222 22273 leader_election.cc:304] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:02.160857 22357 raft_consensus.cc:2804] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:02.163502 22350 tablet_bootstrap.cc:492] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced: No bootstrap required, opened a new log
I20251026 14:58:02.154599 22353 raft_consensus.cc:493] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:02.164052 22350 ts_tablet_manager.cc:1403] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced: Time spent bootstrapping tablet: real 0.108s user 0.043s sys 0.027s
I20251026 14:58:02.166893 22350 raft_consensus.cc:359] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.167713 22350 raft_consensus.cc:385] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:02.167096 22052 catalog_manager.cc:5649] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2). New cstate: current_term: 1 leader_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:02.154891 22364 raft_consensus.cc:493] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:02.168598 22364 raft_consensus.cc:3060] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.168012 22350 raft_consensus.cc:740] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Initialized, Role: FOLLOWER
I20251026 14:58:02.171176 22350 consensus_queue.cc:260] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.173614 22350 ts_tablet_manager.cc:1434] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20251026 14:58:02.174564 22350 tablet_bootstrap.cc:492] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced: Bootstrap starting.
I20251026 14:58:02.176497 22353 raft_consensus.cc:515] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.178673 22353 leader_election.cc:290] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:02.179292 22364 raft_consensus.cc:515] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.180182 22231 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e0f3fab0c2e40daa0e161d82949044d" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" is_pre_election: true
I20251026 14:58:02.181003 22350 tablet_bootstrap.cc:654] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:02.180814 22231 raft_consensus.cc:2468] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 0.
I20251026 14:58:02.181299 22364 leader_election.cc:290] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 election: Requested vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737)
I20251026 14:58:02.182133 22306 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e0f3fab0c2e40daa0e161d82949044d" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bdf853a41f4c14a37d70908b8c2640" is_pre_election: true
I20251026 14:58:02.182979 22306 raft_consensus.cc:2468] T 9e0f3fab0c2e40daa0e161d82949044d P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 0.
I20251026 14:58:02.185184 22158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2cf56c1758324aebad5521cae8b061aa" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
I20251026 14:58:02.186020 22158 raft_consensus.cc:3060] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.188835 22347 tablet_bootstrap.cc:654] T e55b543aef104e7290aa5dae37b421cb P f6b6590ae6a440d1bcb913208797d2ea: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:02.190130 22123 leader_election.cc:304] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:02.192320 22231 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2cf56c1758324aebad5521cae8b061aa" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
I20251026 14:58:02.192958 22231 raft_consensus.cc:3060] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.197607 22353 raft_consensus.cc:2804] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:02.201452 22353 raft_consensus.cc:493] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:02.201958 22353 raft_consensus.cc:3060] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.204792 22231 raft_consensus.cc:2468] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 1.
I20251026 14:58:02.205657 22357 raft_consensus.cc:697] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Becoming Leader. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Running, Role: LEADER
I20251026 14:58:02.206380 22271 leader_election.cc:304] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, 8b40ffb1f6864b108c15f0b8aefeaced; no voters:
I20251026 14:58:02.206579 22357 consensus_queue.cc:237] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.207477 22158 raft_consensus.cc:2468] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 1.
I20251026 14:58:02.207615 22364 raft_consensus.cc:493] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:02.213600 22364 raft_consensus.cc:515] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.214417 22347 tablet_bootstrap.cc:492] T e55b543aef104e7290aa5dae37b421cb P f6b6590ae6a440d1bcb913208797d2ea: No bootstrap required, opened a new log
I20251026 14:58:02.214931 22347 ts_tablet_manager.cc:1403] T e55b543aef104e7290aa5dae37b421cb P f6b6590ae6a440d1bcb913208797d2ea: Time spent bootstrapping tablet: real 0.062s user 0.019s sys 0.012s
I20251026 14:58:02.216497 22157 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4b102dcc6534e2583371c5b81e2da5b" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea" is_pre_election: true
I20251026 14:58:02.217132 22157 raft_consensus.cc:2468] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 0.
I20251026 14:58:02.219038 22364 leader_election.cc:290] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737)
I20251026 14:58:02.218523 22347 raft_consensus.cc:359] T e55b543aef104e7290aa5dae37b421cb P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.218449 22052 catalog_manager.cc:5649] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "12bdf853a41f4c14a37d70908b8c2640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:02.219555 22347 raft_consensus.cc:385] T e55b543aef104e7290aa5dae37b421cb P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:02.219956 22347 raft_consensus.cc:740] T e55b543aef104e7290aa5dae37b421cb P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Initialized, Role: FOLLOWER
I20251026 14:58:02.218590 22273 leader_election.cc:304] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:02.222466 22231 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4b102dcc6534e2583371c5b81e2da5b" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" is_pre_election: true
I20251026 14:58:02.223230 22350 tablet_bootstrap.cc:492] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced: No bootstrap required, opened a new log
I20251026 14:58:02.212296 22353 raft_consensus.cc:515] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.223562 22364 raft_consensus.cc:2804] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:02.224026 22364 raft_consensus.cc:493] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:02.224462 22364 raft_consensus.cc:3060] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.223695 22350 ts_tablet_manager.cc:1403] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced: Time spent bootstrapping tablet: real 0.049s user 0.017s sys 0.014s
I20251026 14:58:02.223111 22231 raft_consensus.cc:2468] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 0.
I20251026 14:58:02.226753 22231 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e0f3fab0c2e40daa0e161d82949044d" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
I20251026 14:58:02.227399 22231 raft_consensus.cc:3060] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.227155 22350 raft_consensus.cc:359] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.227864 22350 raft_consensus.cc:385] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:02.228154 22350 raft_consensus.cc:740] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Initialized, Role: FOLLOWER
I20251026 14:58:02.228819 22350 consensus_queue.cc:260] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.224119 22347 consensus_queue.cc:260] T e55b543aef104e7290aa5dae37b421cb P f6b6590ae6a440d1bcb913208797d2ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.232976 22347 ts_tablet_manager.cc:1434] T e55b543aef104e7290aa5dae37b421cb P f6b6590ae6a440d1bcb913208797d2ea: Time spent starting tablet: real 0.018s user 0.004s sys 0.004s
I20251026 14:58:02.208132 22392 raft_consensus.cc:2804] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:02.233753 22353 leader_election.cc:290] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 1 election: Requested vote from peers 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:02.234153 22392 raft_consensus.cc:697] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Becoming Leader. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Running, Role: LEADER
I20251026 14:58:02.235239 22306 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e0f3fab0c2e40daa0e161d82949044d" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
I20251026 14:58:02.235087 22392 consensus_queue.cc:237] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.235831 22306 raft_consensus.cc:3060] T 9e0f3fab0c2e40daa0e161d82949044d P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.238512 22350 ts_tablet_manager.cc:1434] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20251026 14:58:02.245990 22231 raft_consensus.cc:2468] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 1.
I20251026 14:58:02.245915 22052 catalog_manager.cc:5649] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "12bdf853a41f4c14a37d70908b8c2640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:02.247279 22122 leader_election.cc:304] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b40ffb1f6864b108c15f0b8aefeaced, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:02.248344 22353 raft_consensus.cc:2804] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:02.248831 22353 raft_consensus.cc:697] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [term 1 LEADER]: Becoming Leader. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Running, Role: LEADER
I20251026 14:58:02.241816 22364 raft_consensus.cc:515] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.249670 22353 consensus_queue.cc:237] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.253808 22158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4b102dcc6534e2583371c5b81e2da5b" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
I20251026 14:58:02.254431 22158 raft_consensus.cc:3060] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.260480 22231 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4b102dcc6534e2583371c5b81e2da5b" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
I20251026 14:58:02.260218 22364 leader_election.cc:290] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 election: Requested vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737)
I20251026 14:58:02.261418 22231 raft_consensus.cc:3060] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.262326 22158 raft_consensus.cc:2468] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 1.
I20251026 14:58:02.263624 22273 leader_election.cc:304] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:02.265663 22306 raft_consensus.cc:2468] T 9e0f3fab0c2e40daa0e161d82949044d P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 1.
I20251026 14:58:02.266597 22364 raft_consensus.cc:2804] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:02.268464 22364 raft_consensus.cc:697] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Becoming Leader. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Running, Role: LEADER
I20251026 14:58:02.268606 22231 raft_consensus.cc:2468] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 1.
I20251026 14:58:02.269564 22364 consensus_queue.cc:237] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.270777 22392 raft_consensus.cc:493] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:02.271253 22392 raft_consensus.cc:515] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.273295 22392 leader_election.cc:290] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737)
I20251026 14:58:02.274546 22231 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a8c1bc474434dd88d5a7eceeeece989" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" is_pre_election: true
I20251026 14:58:02.275028 22158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a8c1bc474434dd88d5a7eceeeece989" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea" is_pre_election: true
I20251026 14:58:02.275202 22231 raft_consensus.cc:2468] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 0.
I20251026 14:58:02.275668 22158 raft_consensus.cc:2468] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 0.
I20251026 14:58:02.276269 22271 leader_election.cc:304] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, 8b40ffb1f6864b108c15f0b8aefeaced; no voters:
I20251026 14:58:02.277299 22392 raft_consensus.cc:2804] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:02.277678 22392 raft_consensus.cc:493] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:02.278015 22392 raft_consensus.cc:3060] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.279229 22053 catalog_manager.cc:5649] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "12bdf853a41f4c14a37d70908b8c2640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:02.279743 22052 catalog_manager.cc:5649] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea reported cstate change: term changed from 0 to 1, leader changed from <none> to f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1). New cstate: current_term: 1 leader_uuid: "f6b6590ae6a440d1bcb913208797d2ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:02.285508 22392 raft_consensus.cc:515] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.288409 22158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a8c1bc474434dd88d5a7eceeeece989" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
I20251026 14:58:02.289013 22158 raft_consensus.cc:3060] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.289687 22231 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a8c1bc474434dd88d5a7eceeeece989" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
I20251026 14:58:02.290261 22231 raft_consensus.cc:3060] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.293502 22392 leader_election.cc:290] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 election: Requested vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737)
I20251026 14:58:02.295575 22158 raft_consensus.cc:2468] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 1.
I20251026 14:58:02.296577 22273 leader_election.cc:304] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:02.297264 22392 raft_consensus.cc:2804] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:02.297827 22392 raft_consensus.cc:697] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Becoming Leader. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Running, Role: LEADER
I20251026 14:58:02.297855 22231 raft_consensus.cc:2468] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 1.
I20251026 14:58:02.298800 22392 consensus_queue.cc:237] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.305979 22052 catalog_manager.cc:5649] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "12bdf853a41f4c14a37d70908b8c2640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:02.327417 22354 raft_consensus.cc:493] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:02.327934 22354 raft_consensus.cc:515] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.329478 22354 leader_election.cc:290] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:02.330322 22158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e79d8a8b9394af5b39918f8a0a43b39" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea" is_pre_election: true
I20251026 14:58:02.330408 22306 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e79d8a8b9394af5b39918f8a0a43b39" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bdf853a41f4c14a37d70908b8c2640" is_pre_election: true
I20251026 14:58:02.330873 22158 raft_consensus.cc:2468] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b40ffb1f6864b108c15f0b8aefeaced in term 0.
I20251026 14:58:02.330946 22306 raft_consensus.cc:2468] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b40ffb1f6864b108c15f0b8aefeaced in term 0.
I20251026 14:58:02.331856 22354 raft_consensus.cc:493] T 638b4cb4614f49578bd04fc043b057f3 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:02.331856 22199 leader_election.cc:304] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b40ffb1f6864b108c15f0b8aefeaced, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:02.332455 22354 raft_consensus.cc:515] T 638b4cb4614f49578bd04fc043b057f3 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.332820 22355 raft_consensus.cc:2804] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:02.333199 22355 raft_consensus.cc:493] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:02.333505 22355 raft_consensus.cc:3060] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.334362 22354 leader_election.cc:290] T 638b4cb4614f49578bd04fc043b057f3 P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:02.335157 22158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "638b4cb4614f49578bd04fc043b057f3" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea" is_pre_election: true
I20251026 14:58:02.335843 22158 raft_consensus.cc:2393] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8b40ffb1f6864b108c15f0b8aefeaced in current term 1: Already voted for candidate 12bdf853a41f4c14a37d70908b8c2640 in this term.
I20251026 14:58:02.335750 22306 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "638b4cb4614f49578bd04fc043b057f3" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bdf853a41f4c14a37d70908b8c2640" is_pre_election: true
I20251026 14:58:02.337057 22198 leader_election.cc:304] T 638b4cb4614f49578bd04fc043b057f3 P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b40ffb1f6864b108c15f0b8aefeaced; no voters: 12bdf853a41f4c14a37d70908b8c2640, f6b6590ae6a440d1bcb913208797d2ea
I20251026 14:58:02.337739 22354 raft_consensus.cc:3060] T 638b4cb4614f49578bd04fc043b057f3 P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.339224 22355 raft_consensus.cc:515] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.340921 22355 leader_election.cc:290] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 1 election: Requested vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:02.341758 22158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e79d8a8b9394af5b39918f8a0a43b39" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
I20251026 14:58:02.342073 22306 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e79d8a8b9394af5b39918f8a0a43b39" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
I20251026 14:58:02.342378 22158 raft_consensus.cc:3060] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.342609 22306 raft_consensus.cc:3060] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.342809 22354 raft_consensus.cc:2749] T 638b4cb4614f49578bd04fc043b057f3 P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:02.349429 22306 raft_consensus.cc:2468] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b40ffb1f6864b108c15f0b8aefeaced in term 1.
I20251026 14:58:02.349429 22158 raft_consensus.cc:2468] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b40ffb1f6864b108c15f0b8aefeaced in term 1.
I20251026 14:58:02.350747 22199 leader_election.cc:304] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b40ffb1f6864b108c15f0b8aefeaced, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:02.351441 22354 raft_consensus.cc:2804] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:02.351857 22354 raft_consensus.cc:697] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 LEADER]: Becoming Leader. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Running, Role: LEADER
I20251026 14:58:02.352459 22354 consensus_queue.cc:237] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.360078 22052 catalog_manager.cc:5649] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2). New cstate: current_term: 1 leader_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:02.369833 22392 consensus_queue.cc:1048] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:02.373814 22354 raft_consensus.cc:493] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:02.374300 22354 raft_consensus.cc:515] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.376072 22354 leader_election.cc:290] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:02.376845 22158 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e55b543aef104e7290aa5dae37b421cb" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea" is_pre_election: true
I20251026 14:58:02.377521 22158 raft_consensus.cc:2468] T e55b543aef104e7290aa5dae37b421cb P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b40ffb1f6864b108c15f0b8aefeaced in term 0.
I20251026 14:58:02.377825 22306 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e55b543aef104e7290aa5dae37b421cb" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bdf853a41f4c14a37d70908b8c2640" is_pre_election: true
I20251026 14:58:02.378432 22306 raft_consensus.cc:2468] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b40ffb1f6864b108c15f0b8aefeaced in term 0.
I20251026 14:58:02.378861 22199 leader_election.cc:304] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b40ffb1f6864b108c15f0b8aefeaced, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:02.379714 22354 raft_consensus.cc:2804] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:02.380102 22354 raft_consensus.cc:493] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:02.380475 22354 raft_consensus.cc:3060] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.385151 22357 consensus_queue.cc:1048] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251026 14:58:02.393164 22354 raft_consensus.cc:515] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.396664 22157 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e55b543aef104e7290aa5dae37b421cb" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
I20251026 14:58:02.397188 22157 raft_consensus.cc:3060] T e55b543aef104e7290aa5dae37b421cb P f6b6590ae6a440d1bcb913208797d2ea [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.398093 22306 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e55b543aef104e7290aa5dae37b421cb" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
I20251026 14:58:02.398705 22306 raft_consensus.cc:3060] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:02.401222 22354 leader_election.cc:290] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 1 election: Requested vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:02.404814 22157 raft_consensus.cc:2468] T e55b543aef104e7290aa5dae37b421cb P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b40ffb1f6864b108c15f0b8aefeaced in term 1.
I20251026 14:58:02.406116 22199 leader_election.cc:304] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b40ffb1f6864b108c15f0b8aefeaced, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:02.406705 22306 raft_consensus.cc:2468] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b40ffb1f6864b108c15f0b8aefeaced in term 1.
I20251026 14:58:02.406864 22354 raft_consensus.cc:2804] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:02.407521 22354 raft_consensus.cc:697] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 LEADER]: Becoming Leader. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Running, Role: LEADER
W20251026 14:58:02.408074 22259 tablet.cc:2378] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 14:58:02.408457 22354 consensus_queue.cc:237] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.416214 22052 catalog_manager.cc:5649] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2). New cstate: current_term: 1 leader_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:02.523766 22353 consensus_queue.cc:1048] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:02.533991 22408 consensus_queue.cc:1048] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:02.615799 22354 consensus_queue.cc:1048] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:02.626142 22354 consensus_queue.cc:1048] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:02.652396 22231 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "304dea4666704f0c81d11c8ed58593cc"
dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
mode: GRACEFUL
new_leader_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
from {username='slave'} at 127.0.0.1:53752
I20251026 14:58:02.652982 22231 raft_consensus.cc:606] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 LEADER]: Received request to transfer leadership to TS f6b6590ae6a440d1bcb913208797d2ea
I20251026 14:58:02.657236 22408 consensus_queue.cc:1048] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:02.658108 22357 consensus_queue.cc:1048] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:02.668155 22357 consensus_queue.cc:1048] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:02.674659 22306 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "638b4cb4614f49578bd04fc043b057f3"
dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
mode: GRACEFUL
new_leader_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
from {username='slave'} at 127.0.0.1:33834
I20251026 14:58:02.675628 22306 raft_consensus.cc:606] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Received request to transfer leadership to TS f6b6590ae6a440d1bcb913208797d2ea
I20251026 14:58:02.679497 22408 consensus_queue.cc:1048] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:02.693545 22307 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2cf56c1758324aebad5521cae8b061aa"
dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
mode: GRACEFUL
new_leader_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
from {username='slave'} at 127.0.0.1:33834
I20251026 14:58:02.694180 22307 raft_consensus.cc:606] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Received request to transfer leadership to TS f6b6590ae6a440d1bcb913208797d2ea
I20251026 14:58:02.693583 22357 consensus_queue.cc:1048] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20251026 14:58:02.698356 22392 raft_consensus.cc:993] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640: : Instructing follower f6b6590ae6a440d1bcb913208797d2ea to start an election
I20251026 14:58:02.698848 22392 raft_consensus.cc:1081] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Signalling peer f6b6590ae6a440d1bcb913208797d2ea to start an election
I20251026 14:58:02.702101 22156 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2cf56c1758324aebad5521cae8b061aa"
dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
from {username='slave'} at 127.0.0.1:60116
I20251026 14:58:02.702744 22156 raft_consensus.cc:493] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:58:02.703126 22156 raft_consensus.cc:3060] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:02.707072 22306 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "da9afe4ffb9547c0b7ff3ed93b58248f"
dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
mode: GRACEFUL
new_leader_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
from {username='slave'} at 127.0.0.1:33834
I20251026 14:58:02.707875 22306 raft_consensus.cc:606] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Received request to transfer leadership to TS f6b6590ae6a440d1bcb913208797d2ea
I20251026 14:58:02.711393 22392 consensus_queue.cc:1048] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:02.719219 22156 raft_consensus.cc:515] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.721453 22156 leader_election.cc:290] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 2 election: Requested vote from peers 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:02.721570 22157 raft_consensus.cc:1240] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Rejecting Update request from peer 12bdf853a41f4c14a37d70908b8c2640 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:02.723727 22307 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2cf56c1758324aebad5521cae8b061aa" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
I20251026 14:58:02.724395 22307 raft_consensus.cc:3055] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:02.724733 22307 raft_consensus.cc:740] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Running, Role: LEADER
I20251026 14:58:02.725236 22232 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2cf56c1758324aebad5521cae8b061aa" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
I20251026 14:58:02.726012 22232 raft_consensus.cc:3060] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:02.725545 22307 consensus_queue.cc:260] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.726406 22306 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4a8c1bc474434dd88d5a7eceeeece989"
dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
mode: GRACEFUL
new_leader_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
from {username='slave'} at 127.0.0.1:33834
I20251026 14:58:02.727028 22306 raft_consensus.cc:606] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Received request to transfer leadership to TS f6b6590ae6a440d1bcb913208797d2ea
I20251026 14:58:02.727082 22307 raft_consensus.cc:3060] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:02.728353 22392 consensus_queue.cc:1048] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251026 14:58:02.738229 22392 raft_consensus.cc:993] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640: : Instructing follower f6b6590ae6a440d1bcb913208797d2ea to start an election
I20251026 14:58:02.738823 22392 raft_consensus.cc:1081] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Signalling peer f6b6590ae6a440d1bcb913208797d2ea to start an election
I20251026 14:58:02.743078 22364 consensus_queue.cc:1048] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:02.743309 22232 raft_consensus.cc:2468] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 2.
I20251026 14:58:02.744558 22122 leader_election.cc:304] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b40ffb1f6864b108c15f0b8aefeaced, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:02.745424 22408 raft_consensus.cc:2804] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:02.745908 22408 raft_consensus.cc:697] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Becoming Leader. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Running, Role: LEADER
I20251026 14:58:02.734782 22307 raft_consensus.cc:2468] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 2.
I20251026 14:58:02.747269 22408 consensus_queue.cc:237] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.748332 22158 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "da9afe4ffb9547c0b7ff3ed93b58248f"
dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
from {username='slave'} at 127.0.0.1:60116
I20251026 14:58:02.748900 22158 raft_consensus.cc:493] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:58:02.749257 22158 raft_consensus.cc:3060] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:02.753540 22306 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f4b102dcc6534e2583371c5b81e2da5b"
dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
mode: GRACEFUL
new_leader_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
from {username='slave'} at 127.0.0.1:33834
I20251026 14:58:02.754269 22306 raft_consensus.cc:606] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Received request to transfer leadership to TS 8b40ffb1f6864b108c15f0b8aefeaced
I20251026 14:58:02.755491 22357 raft_consensus.cc:993] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640: : Instructing follower 8b40ffb1f6864b108c15f0b8aefeaced to start an election
I20251026 14:58:02.756042 22392 raft_consensus.cc:1081] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Signalling peer 8b40ffb1f6864b108c15f0b8aefeaced to start an election
I20251026 14:58:02.760183 22053 catalog_manager.cc:5649] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea reported cstate change: term changed from 1 to 2, leader changed from 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3) to f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1). New cstate: current_term: 2 leader_uuid: "f6b6590ae6a440d1bcb913208797d2ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:02.769484 22158 raft_consensus.cc:515] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.769773 22233 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "f4b102dcc6534e2583371c5b81e2da5b"
dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
from {username='slave'} at 127.0.0.1:53746
I20251026 14:58:02.779476 22392 consensus_queue.cc:1048] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:02.792778 22231 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da9afe4ffb9547c0b7ff3ed93b58248f" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
I20251026 14:58:02.774739 22364 consensus_queue.cc:1048] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:02.793540 22231 raft_consensus.cc:3060] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:02.794176 22233 raft_consensus.cc:493] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:58:02.796494 22233 raft_consensus.cc:3060] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:02.805675 22155 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c7ad349a3a2c4e6e8235a080aab372b2"
dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
mode: GRACEFUL
new_leader_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
from {username='slave'} at 127.0.0.1:60118
I20251026 14:58:02.806269 22155 raft_consensus.cc:606] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 1 LEADER]: Received request to transfer leadership to TS 8b40ffb1f6864b108c15f0b8aefeaced
I20251026 14:58:02.809053 22306 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da9afe4ffb9547c0b7ff3ed93b58248f" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
I20251026 14:58:02.809741 22306 raft_consensus.cc:3055] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:02.810047 22306 raft_consensus.cc:740] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Running, Role: LEADER
I20251026 14:58:02.810837 22306 consensus_queue.cc:260] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.811190 22233 raft_consensus.cc:515] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.812011 22306 raft_consensus.cc:3060] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:02.813652 22155 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9e0f3fab0c2e40daa0e161d82949044d"
dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
mode: GRACEFUL
new_leader_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
from {username='slave'} at 127.0.0.1:60118
I20251026 14:58:02.813918 22158 leader_election.cc:290] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 2 election: Requested vote from peers 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:02.814957 22154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4b102dcc6534e2583371c5b81e2da5b" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
I20251026 14:58:02.814965 22233 leader_election.cc:290] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 2 election: Requested vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:02.817490 22231 raft_consensus.cc:2468] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 2.
I20251026 14:58:02.817891 22307 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4b102dcc6534e2583371c5b81e2da5b" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
I20251026 14:58:02.819615 22307 raft_consensus.cc:3055] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:02.819968 22307 raft_consensus.cc:740] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Running, Role: LEADER
I20251026 14:58:02.820580 22122 leader_election.cc:304] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b40ffb1f6864b108c15f0b8aefeaced, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:02.820776 22307 consensus_queue.cc:260] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.821488 22408 raft_consensus.cc:2804] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:02.821537 22411 consensus_queue.cc:1048] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:02.821904 22307 raft_consensus.cc:3060] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:02.822049 22408 raft_consensus.cc:697] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Becoming Leader. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Running, Role: LEADER
I20251026 14:58:02.822954 22408 consensus_queue.cc:237] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.827437 22155 raft_consensus.cc:606] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [term 1 LEADER]: Received request to transfer leadership to TS 8b40ffb1f6864b108c15f0b8aefeaced
I20251026 14:58:02.829226 22307 raft_consensus.cc:2468] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b40ffb1f6864b108c15f0b8aefeaced in term 2.
I20251026 14:58:02.830473 22198 leader_election.cc:304] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, 8b40ffb1f6864b108c15f0b8aefeaced; no voters: f6b6590ae6a440d1bcb913208797d2ea
I20251026 14:58:02.831733 22411 raft_consensus.cc:2804] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:02.832180 22411 raft_consensus.cc:697] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 LEADER]: Becoming Leader. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Running, Role: LEADER
I20251026 14:58:02.833024 22411 consensus_queue.cc:237] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.839048 22392 raft_consensus.cc:993] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640: : Instructing follower f6b6590ae6a440d1bcb913208797d2ea to start an election
I20251026 14:58:02.839689 22392 raft_consensus.cc:1081] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Signalling peer f6b6590ae6a440d1bcb913208797d2ea to start an election
I20251026 14:58:02.841188 22155 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "4a8c1bc474434dd88d5a7eceeeece989"
dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
from {username='slave'} at 127.0.0.1:60116
I20251026 14:58:02.842799 22232 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "e55b543aef104e7290aa5dae37b421cb"
dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
mode: GRACEFUL
new_leader_uuid: "12bdf853a41f4c14a37d70908b8c2640"
from {username='slave'} at 127.0.0.1:53752
I20251026 14:58:02.843546 22232 raft_consensus.cc:606] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 LEADER]: Received request to transfer leadership to TS 12bdf853a41f4c14a37d70908b8c2640
I20251026 14:58:02.841809 22155 raft_consensus.cc:493] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:58:02.845433 22155 raft_consensus.cc:3060] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:02.848668 22053 catalog_manager.cc:5649] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced reported cstate change: term changed from 1 to 2, leader changed from 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3) to 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2). New cstate: current_term: 2 leader_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:02.856881 22306 raft_consensus.cc:2468] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 2.
I20251026 14:58:02.859203 22357 raft_consensus.cc:993] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640: : Instructing follower f6b6590ae6a440d1bcb913208797d2ea to start an election
I20251026 14:58:02.859758 22364 raft_consensus.cc:1081] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Signalling peer f6b6590ae6a440d1bcb913208797d2ea to start an election
I20251026 14:58:02.859738 22411 consensus_queue.cc:1048] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:02.861992 22157 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "638b4cb4614f49578bd04fc043b057f3"
dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
from {username='slave'} at 127.0.0.1:60116
I20251026 14:58:02.862527 22157 raft_consensus.cc:493] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:58:02.862896 22157 raft_consensus.cc:3060] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:02.866674 22155 raft_consensus.cc:515] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.870965 22232 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a8c1bc474434dd88d5a7eceeeece989" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
I20251026 14:58:02.871616 22232 raft_consensus.cc:3060] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:02.873968 22305 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a8c1bc474434dd88d5a7eceeeece989" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
I20251026 14:58:02.874616 22305 raft_consensus.cc:3055] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:02.874924 22305 raft_consensus.cc:740] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Running, Role: LEADER
I20251026 14:58:02.875186 22156 raft_consensus.cc:1240] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Rejecting Update request from peer 12bdf853a41f4c14a37d70908b8c2640 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:02.875746 22305 consensus_queue.cc:260] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.878551 22305 raft_consensus.cc:3060] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:02.882093 22052 catalog_manager.cc:5649] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea reported cstate change: term changed from 1 to 2, leader changed from 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3) to f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1). New cstate: current_term: 2 leader_uuid: "f6b6590ae6a440d1bcb913208797d2ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:02.886539 22305 raft_consensus.cc:2468] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 2.
I20251026 14:58:02.887563 22157 raft_consensus.cc:515] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.887110 22155 leader_election.cc:290] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 2 election: Requested vote from peers 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:02.890475 22232 raft_consensus.cc:2468] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 2.
I20251026 14:58:02.893683 22123 leader_election.cc:304] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:02.894193 22232 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "638b4cb4614f49578bd04fc043b057f3" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
I20251026 14:58:02.894622 22408 raft_consensus.cc:2804] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:02.894841 22232 raft_consensus.cc:3060] T 638b4cb4614f49578bd04fc043b057f3 P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:02.895171 22408 raft_consensus.cc:697] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Becoming Leader. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Running, Role: LEADER
I20251026 14:58:02.896013 22408 consensus_queue.cc:237] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.901484 22157 leader_election.cc:290] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 2 election: Requested vote from peers 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:02.902570 22305 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "638b4cb4614f49578bd04fc043b057f3" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
I20251026 14:58:02.903107 22305 raft_consensus.cc:3055] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:02.903345 22305 raft_consensus.cc:740] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Running, Role: LEADER
I20251026 14:58:02.903867 22232 raft_consensus.cc:2468] T 638b4cb4614f49578bd04fc043b057f3 P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 2.
I20251026 14:58:02.904018 22305 consensus_queue.cc:260] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.905100 22305 raft_consensus.cc:3060] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:02.904963 22122 leader_election.cc:304] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b40ffb1f6864b108c15f0b8aefeaced, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:02.905886 22408 raft_consensus.cc:2804] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:02.906373 22408 raft_consensus.cc:697] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Becoming Leader. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Running, Role: LEADER
I20251026 14:58:02.907877 22408 consensus_queue.cc:237] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:02.911067 22052 catalog_manager.cc:5649] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea reported cstate change: term changed from 1 to 2, leader changed from 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3) to f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1). New cstate: current_term: 2 leader_uuid: "f6b6590ae6a440d1bcb913208797d2ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:02.914492 22305 raft_consensus.cc:2468] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 2.
I20251026 14:58:02.924592 22052 catalog_manager.cc:5649] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea reported cstate change: term changed from 1 to 2, leader changed from 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3) to f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1). New cstate: current_term: 2 leader_uuid: "f6b6590ae6a440d1bcb913208797d2ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:02.996743 22429 consensus_queue.cc:1048] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:03.003468 22411 raft_consensus.cc:993] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced: : Instructing follower 12bdf853a41f4c14a37d70908b8c2640 to start an election
I20251026 14:58:03.003957 22354 raft_consensus.cc:1081] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 LEADER]: Signalling peer 12bdf853a41f4c14a37d70908b8c2640 to start an election
I20251026 14:58:03.006182 22306 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "e55b543aef104e7290aa5dae37b421cb"
dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
from {username='slave'} at 127.0.0.1:33826
I20251026 14:58:03.006762 22306 raft_consensus.cc:493] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:58:03.007124 22306 raft_consensus.cc:3060] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:03.008504 22429 consensus_queue.cc:1048] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:03.014212 22306 raft_consensus.cc:515] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:03.015110 22305 raft_consensus.cc:1240] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Rejecting Update request from peer 8b40ffb1f6864b108c15f0b8aefeaced for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:03.016672 22306 leader_election.cc:290] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 2 election: Requested vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737)
I20251026 14:58:03.017992 22154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e55b543aef104e7290aa5dae37b421cb" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
I20251026 14:58:03.018196 22429 consensus_queue.cc:1059] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:03.018431 22232 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e55b543aef104e7290aa5dae37b421cb" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
I20251026 14:58:03.018791 22154 raft_consensus.cc:3060] T e55b543aef104e7290aa5dae37b421cb P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:03.018994 22429 raft_consensus.cc:3055] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:03.019369 22429 raft_consensus.cc:740] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Running, Role: LEADER
I20251026 14:58:03.020262 22429 consensus_queue.cc:260] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:03.021647 22429 raft_consensus.cc:3060] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:03.027177 22154 raft_consensus.cc:2468] T e55b543aef104e7290aa5dae37b421cb P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 2.
I20251026 14:58:03.028584 22273 leader_election.cc:304] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:03.029538 22419 raft_consensus.cc:2804] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:03.030018 22419 raft_consensus.cc:697] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [term 2 LEADER]: Becoming Leader. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Running, Role: LEADER
I20251026 14:58:03.030875 22419 consensus_queue.cc:237] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:03.035594 22232 raft_consensus.cc:2468] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 2.
I20251026 14:58:03.037928 22052 catalog_manager.cc:5649] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 reported cstate change: term changed from 1 to 2, leader changed from 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2) to 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3). New cstate: current_term: 2 leader_uuid: "12bdf853a41f4c14a37d70908b8c2640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:03.087910 22355 raft_consensus.cc:993] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced: : Instructing follower f6b6590ae6a440d1bcb913208797d2ea to start an election
I20251026 14:58:03.088354 22429 raft_consensus.cc:1081] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 LEADER]: Signalling peer f6b6590ae6a440d1bcb913208797d2ea to start an election
I20251026 14:58:03.089699 22154 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "304dea4666704f0c81d11c8ed58593cc"
dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
from {username='slave'} at 127.0.0.1:60108
I20251026 14:58:03.090163 22154 raft_consensus.cc:493] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:58:03.090438 22154 raft_consensus.cc:3060] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:03.095180 22154 raft_consensus.cc:515] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:03.096781 22154 leader_election.cc:290] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 2 election: Requested vote from peers 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:03.097994 22306 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "304dea4666704f0c81d11c8ed58593cc" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
I20251026 14:58:03.097987 22232 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "304dea4666704f0c81d11c8ed58593cc" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
I20251026 14:58:03.098511 22306 raft_consensus.cc:3060] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:03.098568 22232 raft_consensus.cc:3055] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:03.098912 22232 raft_consensus.cc:740] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Running, Role: LEADER
I20251026 14:58:03.099534 22232 consensus_queue.cc:260] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:03.100452 22232 raft_consensus.cc:3060] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:03.103308 22306 raft_consensus.cc:2468] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 2.
I20251026 14:58:03.104418 22123 leader_election.cc:304] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:03.104997 22232 raft_consensus.cc:2468] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 2.
I20251026 14:58:03.105070 22408 raft_consensus.cc:2804] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:03.105602 22408 raft_consensus.cc:697] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Becoming Leader. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Running, Role: LEADER
I20251026 14:58:03.106405 22408 consensus_queue.cc:237] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:03.112313 22052 catalog_manager.cc:5649] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea reported cstate change: term changed from 1 to 2, leader changed from 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2) to f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1). New cstate: current_term: 2 leader_uuid: "f6b6590ae6a440d1bcb913208797d2ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:03.161890 22306 raft_consensus.cc:1275] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Refusing update from remote peer f6b6590ae6a440d1bcb913208797d2ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:03.163637 22390 raft_consensus.cc:993] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea: : Instructing follower 8b40ffb1f6864b108c15f0b8aefeaced to start an election
I20251026 14:58:03.163949 22408 consensus_queue.cc:1048] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:03.164263 22390 raft_consensus.cc:1081] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 1 LEADER]: Signalling peer 8b40ffb1f6864b108c15f0b8aefeaced to start an election
I20251026 14:58:03.166085 22232 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c7ad349a3a2c4e6e8235a080aab372b2"
dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
from {username='slave'} at 127.0.0.1:53740
I20251026 14:58:03.167064 22232 raft_consensus.cc:493] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:58:03.167575 22232 raft_consensus.cc:3060] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:03.174510 22231 raft_consensus.cc:1275] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Refusing update from remote peer f6b6590ae6a440d1bcb913208797d2ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:03.177213 22408 consensus_queue.cc:1048] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:58:03.180894 22232 raft_consensus.cc:515] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:03.183497 22232 leader_election.cc:290] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 2 election: Requested vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:03.183809 22154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7ad349a3a2c4e6e8235a080aab372b2" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
I20251026 14:58:03.184423 22154 raft_consensus.cc:3055] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:03.184652 22154 raft_consensus.cc:740] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Running, Role: LEADER
I20251026 14:58:03.185237 22154 consensus_queue.cc:260] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:03.186118 22154 raft_consensus.cc:3060] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:03.189652 22306 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7ad349a3a2c4e6e8235a080aab372b2" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
I20251026 14:58:03.190267 22306 raft_consensus.cc:3060] T c7ad349a3a2c4e6e8235a080aab372b2 P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:03.192641 22154 raft_consensus.cc:2468] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b40ffb1f6864b108c15f0b8aefeaced in term 2.
I20251026 14:58:03.193859 22199 leader_election.cc:304] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b40ffb1f6864b108c15f0b8aefeaced, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:03.194779 22429 raft_consensus.cc:2804] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:03.195250 22429 raft_consensus.cc:697] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 LEADER]: Becoming Leader. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Running, Role: LEADER
I20251026 14:58:03.196051 22429 consensus_queue.cc:237] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:03.199507 22306 raft_consensus.cc:2468] T c7ad349a3a2c4e6e8235a080aab372b2 P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b40ffb1f6864b108c15f0b8aefeaced in term 2.
I20251026 14:58:03.207201 22053 catalog_manager.cc:5649] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced reported cstate change: term changed from 1 to 2, leader changed from f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1) to 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2). New cstate: current_term: 2 leader_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:03.221853 22390 raft_consensus.cc:993] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea: : Instructing follower 8b40ffb1f6864b108c15f0b8aefeaced to start an election
I20251026 14:58:03.222270 22408 raft_consensus.cc:1081] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [term 1 LEADER]: Signalling peer 8b40ffb1f6864b108c15f0b8aefeaced to start an election
I20251026 14:58:03.223636 22232 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9e0f3fab0c2e40daa0e161d82949044d"
dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
from {username='slave'} at 127.0.0.1:53740
I20251026 14:58:03.224107 22232 raft_consensus.cc:493] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:58:03.224406 22232 raft_consensus.cc:3060] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:03.230104 22232 raft_consensus.cc:515] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:03.231889 22232 leader_election.cc:290] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 2 election: Requested vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:03.232792 22154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e0f3fab0c2e40daa0e161d82949044d" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
I20251026 14:58:03.232964 22306 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e0f3fab0c2e40daa0e161d82949044d" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
I20251026 14:58:03.233434 22154 raft_consensus.cc:3055] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:03.233629 22306 raft_consensus.cc:3060] T 9e0f3fab0c2e40daa0e161d82949044d P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:03.233830 22154 raft_consensus.cc:740] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Running, Role: LEADER
I20251026 14:58:03.234768 22154 consensus_queue.cc:260] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:03.235723 22154 raft_consensus.cc:3060] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:03.240511 22306 raft_consensus.cc:2468] T 9e0f3fab0c2e40daa0e161d82949044d P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b40ffb1f6864b108c15f0b8aefeaced in term 2.
I20251026 14:58:03.241178 22305 raft_consensus.cc:1240] T 9e0f3fab0c2e40daa0e161d82949044d P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Rejecting Update request from peer f6b6590ae6a440d1bcb913208797d2ea for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:03.241583 22198 leader_election.cc:304] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, 8b40ffb1f6864b108c15f0b8aefeaced; no voters:
I20251026 14:58:03.242317 22429 raft_consensus.cc:2804] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:03.242702 22429 raft_consensus.cc:697] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 LEADER]: Becoming Leader. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Running, Role: LEADER
I20251026 14:58:03.243384 22154 raft_consensus.cc:2468] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b40ffb1f6864b108c15f0b8aefeaced in term 2.
I20251026 14:58:03.243499 22429 consensus_queue.cc:237] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:03.250129 22053 catalog_manager.cc:5649] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced reported cstate change: term changed from 1 to 2, leader changed from f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1) to 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2). New cstate: current_term: 2 leader_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:03.310160 22305 raft_consensus.cc:1275] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Refusing update from remote peer f6b6590ae6a440d1bcb913208797d2ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:03.311302 22390 consensus_queue.cc:1048] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:03.318485 22232 raft_consensus.cc:1275] T 638b4cb4614f49578bd04fc043b057f3 P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Refusing update from remote peer f6b6590ae6a440d1bcb913208797d2ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:03.320400 22433 consensus_queue.cc:1048] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:58:03.326176 22231 raft_consensus.cc:1275] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Refusing update from remote peer f6b6590ae6a440d1bcb913208797d2ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:03.328099 22408 consensus_queue.cc:1048] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:03.337451 22305 raft_consensus.cc:1275] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Refusing update from remote peer f6b6590ae6a440d1bcb913208797d2ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:03.338830 22408 consensus_queue.cc:1048] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:03.356374 22305 raft_consensus.cc:1275] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Refusing update from remote peer 8b40ffb1f6864b108c15f0b8aefeaced: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:03.357666 22429 consensus_queue.cc:1048] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:03.365716 22154 raft_consensus.cc:3060] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:03.372638 22154 raft_consensus.cc:1275] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Refusing update from remote peer 8b40ffb1f6864b108c15f0b8aefeaced: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:03.374267 22354 consensus_queue.cc:1048] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:03.397846 22305 raft_consensus.cc:1275] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Refusing update from remote peer f6b6590ae6a440d1bcb913208797d2ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:03.399116 22408 consensus_queue.cc:1048] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:03.406694 22231 raft_consensus.cc:1275] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Refusing update from remote peer f6b6590ae6a440d1bcb913208797d2ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:03.408495 22408 consensus_queue.cc:1048] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:03.411571 22231 raft_consensus.cc:1275] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Refusing update from remote peer 12bdf853a41f4c14a37d70908b8c2640: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:03.412798 22419 consensus_queue.cc:1048] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:03.422289 22154 raft_consensus.cc:1275] T e55b543aef104e7290aa5dae37b421cb P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Refusing update from remote peer 12bdf853a41f4c14a37d70908b8c2640: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:03.423841 22434 consensus_queue.cc:1048] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:03.543167 22232 raft_consensus.cc:1275] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Refusing update from remote peer f6b6590ae6a440d1bcb913208797d2ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:03.544446 22408 consensus_queue.cc:1048] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:03.552356 22305 raft_consensus.cc:1275] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Refusing update from remote peer f6b6590ae6a440d1bcb913208797d2ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:03.554785 22390 consensus_queue.cc:1048] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:03.675220 22305 raft_consensus.cc:1275] T c7ad349a3a2c4e6e8235a080aab372b2 P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Refusing update from remote peer 8b40ffb1f6864b108c15f0b8aefeaced: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:03.676657 22354 consensus_queue.cc:1048] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:03.685660 22154 raft_consensus.cc:1275] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Refusing update from remote peer 8b40ffb1f6864b108c15f0b8aefeaced: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:03.687197 22411 consensus_queue.cc:1048] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:03.778681 22305 raft_consensus.cc:1275] T 9e0f3fab0c2e40daa0e161d82949044d P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Refusing update from remote peer 8b40ffb1f6864b108c15f0b8aefeaced: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:03.779963 22411 consensus_queue.cc:1048] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:03.787860 22154 raft_consensus.cc:1275] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Refusing update from remote peer 8b40ffb1f6864b108c15f0b8aefeaced: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:03.789585 22429 consensus_queue.cc:1048] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:05.875511 21512 auto_leader_rebalancer-test.cc:343] The leader distribution is
I20251026 14:58:05.876257 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:58:05.893970 22154 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2cf56c1758324aebad5521cae8b061aa"
dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
mode: GRACEFUL
new_leader_uuid: "12bdf853a41f4c14a37d70908b8c2640"
from {username='slave'} at 127.0.0.1:53182
I20251026 14:58:05.894474 22154 raft_consensus.cc:606] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Received request to transfer leadership to TS 12bdf853a41f4c14a37d70908b8c2640
I20251026 14:58:05.896607 22154 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "da9afe4ffb9547c0b7ff3ed93b58248f"
dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
mode: GRACEFUL
new_leader_uuid: "12bdf853a41f4c14a37d70908b8c2640"
from {username='slave'} at 127.0.0.1:53182
I20251026 14:58:05.897040 22154 raft_consensus.cc:606] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Received request to transfer leadership to TS 12bdf853a41f4c14a37d70908b8c2640
I20251026 14:58:05.897954 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251026 14:58:05.898414 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:58:06.060246 22441 raft_consensus.cc:993] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea: : Instructing follower 12bdf853a41f4c14a37d70908b8c2640 to start an election
I20251026 14:58:06.060768 22444 raft_consensus.cc:1081] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Signalling peer 12bdf853a41f4c14a37d70908b8c2640 to start an election
I20251026 14:58:06.062172 22306 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "da9afe4ffb9547c0b7ff3ed93b58248f"
dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
from {username='slave'} at 127.0.0.1:33828
I20251026 14:58:06.062640 22306 raft_consensus.cc:493] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:58:06.062906 22306 raft_consensus.cc:3060] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:06.067857 22306 raft_consensus.cc:515] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:06.069344 22306 leader_election.cc:290] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 3 election: Requested vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737)
I20251026 14:58:06.070307 22154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da9afe4ffb9547c0b7ff3ed93b58248f" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
I20251026 14:58:06.070369 22232 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da9afe4ffb9547c0b7ff3ed93b58248f" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
I20251026 14:58:06.070890 22154 raft_consensus.cc:3055] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:58:06.070941 22232 raft_consensus.cc:3060] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:06.071201 22154 raft_consensus.cc:740] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Running, Role: LEADER
I20251026 14:58:06.071916 22154 consensus_queue.cc:260] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:06.072927 22154 raft_consensus.cc:3060] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:06.076041 22232 raft_consensus.cc:2468] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 3.
I20251026 14:58:06.077224 22271 leader_election.cc:304] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, 8b40ffb1f6864b108c15f0b8aefeaced; no voters:
I20251026 14:58:06.077970 22440 raft_consensus.cc:2804] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:58:06.078016 22154 raft_consensus.cc:2468] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 3.
I20251026 14:58:06.079082 22440 raft_consensus.cc:697] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 3 LEADER]: Becoming Leader. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Running, Role: LEADER
I20251026 14:58:06.079854 22440 consensus_queue.cc:237] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:06.087193 22052 catalog_manager.cc:5649] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 reported cstate change: term changed from 2 to 3, leader changed from f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1) to 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3). New cstate: current_term: 3 leader_uuid: "12bdf853a41f4c14a37d70908b8c2640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:06.143654 22441 raft_consensus.cc:993] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea: : Instructing follower 12bdf853a41f4c14a37d70908b8c2640 to start an election
I20251026 14:58:06.144043 22444 raft_consensus.cc:1081] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Signalling peer 12bdf853a41f4c14a37d70908b8c2640 to start an election
I20251026 14:58:06.145452 22306 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2cf56c1758324aebad5521cae8b061aa"
dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
from {username='slave'} at 127.0.0.1:33828
I20251026 14:58:06.145905 22306 raft_consensus.cc:493] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:58:06.146144 22306 raft_consensus.cc:3060] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:06.150622 22306 raft_consensus.cc:515] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:06.151953 22306 leader_election.cc:290] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 3 election: Requested vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737)
I20251026 14:58:06.152987 22231 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2cf56c1758324aebad5521cae8b061aa" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
I20251026 14:58:06.152988 22154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2cf56c1758324aebad5521cae8b061aa" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
I20251026 14:58:06.153645 22231 raft_consensus.cc:3060] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:06.153709 22154 raft_consensus.cc:3055] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:58:06.154042 22154 raft_consensus.cc:740] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Running, Role: LEADER
I20251026 14:58:06.154676 22154 consensus_queue.cc:260] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:06.155576 22154 raft_consensus.cc:3060] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:06.158473 22231 raft_consensus.cc:2468] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 3.
I20251026 14:58:06.159504 22271 leader_election.cc:304] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, 8b40ffb1f6864b108c15f0b8aefeaced; no voters:
I20251026 14:58:06.160323 22440 raft_consensus.cc:2804] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:58:06.160678 22154 raft_consensus.cc:2468] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 3.
I20251026 14:58:06.160787 22440 raft_consensus.cc:697] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 3 LEADER]: Becoming Leader. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Running, Role: LEADER
I20251026 14:58:06.161666 22440 consensus_queue.cc:237] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:06.168853 22052 catalog_manager.cc:5649] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 reported cstate change: term changed from 2 to 3, leader changed from f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1) to 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3). New cstate: current_term: 3 leader_uuid: "12bdf853a41f4c14a37d70908b8c2640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:06.592970 22154 raft_consensus.cc:1275] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 3 FOLLOWER]: Refusing update from remote peer 12bdf853a41f4c14a37d70908b8c2640: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:58:06.594285 22440 consensus_queue.cc:1048] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:06.616248 22231 raft_consensus.cc:1275] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [term 3 FOLLOWER]: Refusing update from remote peer 12bdf853a41f4c14a37d70908b8c2640: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:58:06.618808 22445 consensus_queue.cc:1048] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:06.634953 22154 raft_consensus.cc:1275] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 3 FOLLOWER]: Refusing update from remote peer 12bdf853a41f4c14a37d70908b8c2640: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:58:06.636068 22445 consensus_queue.cc:1048] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:06.647353 22231 raft_consensus.cc:1275] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [term 3 FOLLOWER]: Refusing update from remote peer 12bdf853a41f4c14a37d70908b8c2640: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:58:06.648840 22445 consensus_queue.cc:1048] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:06.899272 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:58:06.904620 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:58:06.905069 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:58:07.906051 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:58:07.911940 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:58:07.912428 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:58:08.913462 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:58:08.918977 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:58:08.919420 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:58:09.920521 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:58:09.926955 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:58:09.927433 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:58:10.928503 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:58:10.933687 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:58:10.934118 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:58:11.935050 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:58:11.940354 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:58:11.940832 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:58:12.941831 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:58:12.947494 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:58:12.947953 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:58:13.948904 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:58:13.954034 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:58:13.954493 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:58:14.955618 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:58:14.962182 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:58:14.962600 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:58:15.992056 21512 auto_leader_rebalancer-test.cc:362] The leader distribution is
I20251026 14:58:16.003437 22305 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2cf56c1758324aebad5521cae8b061aa"
dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
mode: GRACEFUL
new_leader_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
from {username='slave'} at 127.0.0.1:33834
I20251026 14:58:16.004081 22305 raft_consensus.cc:606] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 3 LEADER]: Received request to transfer leadership to TS f6b6590ae6a440d1bcb913208797d2ea
I20251026 14:58:16.008461 22305 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "da9afe4ffb9547c0b7ff3ed93b58248f"
dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
mode: GRACEFUL
new_leader_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
from {username='slave'} at 127.0.0.1:33834
I20251026 14:58:16.008916 22305 raft_consensus.cc:606] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 3 LEADER]: Received request to transfer leadership to TS f6b6590ae6a440d1bcb913208797d2ea
I20251026 14:58:16.015856 22232 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9e79d8a8b9394af5b39918f8a0a43b39"
dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
mode: GRACEFUL
new_leader_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
from {username='slave'} at 127.0.0.1:53752
I20251026 14:58:16.016307 22232 raft_consensus.cc:606] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 LEADER]: Received request to transfer leadership to TS f6b6590ae6a440d1bcb913208797d2ea
I20251026 14:58:16.020560 22232 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f4b102dcc6534e2583371c5b81e2da5b"
dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
mode: GRACEFUL
new_leader_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
from {username='slave'} at 127.0.0.1:53752
I20251026 14:58:16.020991 22232 raft_consensus.cc:606] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 LEADER]: Received request to transfer leadership to TS f6b6590ae6a440d1bcb913208797d2ea
I20251026 14:58:16.025480 22232 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c7ad349a3a2c4e6e8235a080aab372b2"
dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
mode: GRACEFUL
new_leader_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
from {username='slave'} at 127.0.0.1:53752
I20251026 14:58:16.025997 22232 raft_consensus.cc:606] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 LEADER]: Received request to transfer leadership to TS f6b6590ae6a440d1bcb913208797d2ea
I20251026 14:58:16.070140 22465 raft_consensus.cc:993] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640: : Instructing follower f6b6590ae6a440d1bcb913208797d2ea to start an election
I20251026 14:58:16.070659 22470 raft_consensus.cc:1081] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 3 LEADER]: Signalling peer f6b6590ae6a440d1bcb913208797d2ea to start an election
I20251026 14:58:16.072166 22157 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2cf56c1758324aebad5521cae8b061aa"
dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
from {username='slave'} at 127.0.0.1:60116
I20251026 14:58:16.072644 22157 raft_consensus.cc:493] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:58:16.073029 22157 raft_consensus.cc:3060] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:58:16.078920 22157 raft_consensus.cc:515] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:16.080809 22465 raft_consensus.cc:993] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640: : Instructing follower f6b6590ae6a440d1bcb913208797d2ea to start an election
I20251026 14:58:16.080950 22157 leader_election.cc:290] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 4 election: Requested vote from peers 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:16.081350 22470 raft_consensus.cc:1081] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 3 LEADER]: Signalling peer f6b6590ae6a440d1bcb913208797d2ea to start an election
I20251026 14:58:16.081974 22232 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2cf56c1758324aebad5521cae8b061aa" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
I20251026 14:58:16.082755 22232 raft_consensus.cc:3060] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:58:16.083192 22154 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "da9afe4ffb9547c0b7ff3ed93b58248f"
dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
from {username='slave'} at 127.0.0.1:60116
I20251026 14:58:16.083555 22154 raft_consensus.cc:493] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:58:16.083746 22154 raft_consensus.cc:3060] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:58:16.082020 22305 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2cf56c1758324aebad5521cae8b061aa" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
I20251026 14:58:16.086452 22305 raft_consensus.cc:3055] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:58:16.086762 22305 raft_consensus.cc:740] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Running, Role: LEADER
I20251026 14:58:16.087654 22232 raft_consensus.cc:2468] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 4.
I20251026 14:58:16.087527 22305 consensus_queue.cc:260] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:16.088901 22305 raft_consensus.cc:3060] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:58:16.088966 22122 leader_election.cc:304] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b40ffb1f6864b108c15f0b8aefeaced, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:16.089874 22444 raft_consensus.cc:2804] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:58:16.089702 22154 raft_consensus.cc:515] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:16.091005 22444 raft_consensus.cc:697] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 4 LEADER]: Becoming Leader. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Running, Role: LEADER
I20251026 14:58:16.091935 22444 consensus_queue.cc:237] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:16.092427 22232 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da9afe4ffb9547c0b7ff3ed93b58248f" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
I20251026 14:58:16.092767 22154 leader_election.cc:290] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 4 election: Requested vote from peers 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:16.092926 22232 raft_consensus.cc:3060] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:58:16.094329 22306 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da9afe4ffb9547c0b7ff3ed93b58248f" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
I20251026 14:58:16.094758 22306 raft_consensus.cc:3055] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:58:16.095016 22306 raft_consensus.cc:740] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Running, Role: LEADER
I20251026 14:58:16.095906 22305 raft_consensus.cc:2468] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 4.
I20251026 14:58:16.095714 22306 consensus_queue.cc:260] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:16.097506 22306 raft_consensus.cc:3060] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:58:16.099447 22232 raft_consensus.cc:2468] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 4.
I20251026 14:58:16.100323 22122 leader_election.cc:304] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b40ffb1f6864b108c15f0b8aefeaced, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:16.101023 22444 raft_consensus.cc:2804] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:58:16.101456 22444 raft_consensus.cc:697] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 4 LEADER]: Becoming Leader. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Running, Role: LEADER
I20251026 14:58:16.102319 22444 consensus_queue.cc:237] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:16.102540 22054 catalog_manager.cc:5649] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea reported cstate change: term changed from 3 to 4, leader changed from 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3) to f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1). New cstate: current_term: 4 leader_uuid: "f6b6590ae6a440d1bcb913208797d2ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:16.104390 22306 raft_consensus.cc:2468] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 4.
I20251026 14:58:16.119270 22478 raft_consensus.cc:993] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced: : Instructing follower f6b6590ae6a440d1bcb913208797d2ea to start an election
I20251026 14:58:16.119793 22464 raft_consensus.cc:1081] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 LEADER]: Signalling peer f6b6590ae6a440d1bcb913208797d2ea to start an election
I20251026 14:58:16.120250 22054 catalog_manager.cc:5649] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea reported cstate change: term changed from 3 to 4, leader changed from 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3) to f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1). New cstate: current_term: 4 leader_uuid: "f6b6590ae6a440d1bcb913208797d2ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:16.121304 22154 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9e79d8a8b9394af5b39918f8a0a43b39"
dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
from {username='slave'} at 127.0.0.1:60108
I20251026 14:58:16.121721 22154 raft_consensus.cc:493] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:58:16.121974 22154 raft_consensus.cc:3060] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:16.127344 22154 raft_consensus.cc:515] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:16.129376 22154 leader_election.cc:290] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 2 election: Requested vote from peers 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:16.129730 22232 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e79d8a8b9394af5b39918f8a0a43b39" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
I20251026 14:58:16.129884 22306 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e79d8a8b9394af5b39918f8a0a43b39" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
I20251026 14:58:16.130177 22232 raft_consensus.cc:3055] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:16.130311 22306 raft_consensus.cc:3060] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:16.130457 22232 raft_consensus.cc:740] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Running, Role: LEADER
I20251026 14:58:16.131129 22232 consensus_queue.cc:260] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:16.131891 22232 raft_consensus.cc:3060] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:16.134529 22306 raft_consensus.cc:2468] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 2.
I20251026 14:58:16.135766 22123 leader_election.cc:304] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:16.136339 22232 raft_consensus.cc:2468] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 2.
I20251026 14:58:16.136590 22444 raft_consensus.cc:2804] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:16.137035 22444 raft_consensus.cc:697] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Becoming Leader. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Running, Role: LEADER
I20251026 14:58:16.137804 22444 consensus_queue.cc:237] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:16.145318 22054 catalog_manager.cc:5649] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea reported cstate change: term changed from 1 to 2, leader changed from 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2) to f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1). New cstate: current_term: 2 leader_uuid: "f6b6590ae6a440d1bcb913208797d2ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:16.193873 22478 raft_consensus.cc:993] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced: : Instructing follower f6b6590ae6a440d1bcb913208797d2ea to start an election
I20251026 14:58:16.194357 22464 raft_consensus.cc:1081] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 LEADER]: Signalling peer f6b6590ae6a440d1bcb913208797d2ea to start an election
I20251026 14:58:16.195837 22154 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "f4b102dcc6534e2583371c5b81e2da5b"
dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
from {username='slave'} at 127.0.0.1:60108
I20251026 14:58:16.196264 22154 raft_consensus.cc:493] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:58:16.196487 22154 raft_consensus.cc:3060] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:16.201321 22154 raft_consensus.cc:515] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:16.202702 22154 leader_election.cc:290] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 3 election: Requested vote from peers 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:16.203467 22232 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4b102dcc6534e2583371c5b81e2da5b" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
I20251026 14:58:16.203682 22306 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4b102dcc6534e2583371c5b81e2da5b" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
I20251026 14:58:16.203960 22232 raft_consensus.cc:3055] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:58:16.204128 22306 raft_consensus.cc:3060] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:16.204238 22232 raft_consensus.cc:740] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Running, Role: LEADER
I20251026 14:58:16.204913 22232 consensus_queue.cc:260] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:16.205974 22232 raft_consensus.cc:3060] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:16.208514 22306 raft_consensus.cc:2468] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 3.
I20251026 14:58:16.209434 22123 leader_election.cc:304] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:16.210228 22471 raft_consensus.cc:2804] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:58:16.210223 22232 raft_consensus.cc:2468] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 3.
I20251026 14:58:16.210776 22471 raft_consensus.cc:697] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea [term 3 LEADER]: Becoming Leader. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Running, Role: LEADER
I20251026 14:58:16.211709 22471 consensus_queue.cc:237] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:16.218488 22054 catalog_manager.cc:5649] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea reported cstate change: term changed from 2 to 3, leader changed from 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2) to f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1). New cstate: current_term: 3 leader_uuid: "f6b6590ae6a440d1bcb913208797d2ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:16.293983 22478 raft_consensus.cc:993] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced: : Instructing follower f6b6590ae6a440d1bcb913208797d2ea to start an election
I20251026 14:58:16.294430 22478 raft_consensus.cc:1081] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 LEADER]: Signalling peer f6b6590ae6a440d1bcb913208797d2ea to start an election
I20251026 14:58:16.295826 22154 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c7ad349a3a2c4e6e8235a080aab372b2"
dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
from {username='slave'} at 127.0.0.1:60108
I20251026 14:58:16.296262 22154 raft_consensus.cc:493] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:58:16.296526 22154 raft_consensus.cc:3060] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:16.300622 22154 raft_consensus.cc:515] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:16.301970 22154 leader_election.cc:290] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 3 election: Requested vote from peers 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:16.302927 22232 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7ad349a3a2c4e6e8235a080aab372b2" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
I20251026 14:58:16.302996 22306 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7ad349a3a2c4e6e8235a080aab372b2" candidate_uuid: "f6b6590ae6a440d1bcb913208797d2ea" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
I20251026 14:58:16.303437 22232 raft_consensus.cc:3055] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:58:16.303488 22306 raft_consensus.cc:3060] T c7ad349a3a2c4e6e8235a080aab372b2 P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:16.303705 22232 raft_consensus.cc:740] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Running, Role: LEADER
I20251026 14:58:16.304316 22232 consensus_queue.cc:260] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:16.305119 22232 raft_consensus.cc:3060] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:16.308295 22306 raft_consensus.cc:2468] T c7ad349a3a2c4e6e8235a080aab372b2 P 12bdf853a41f4c14a37d70908b8c2640 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 3.
I20251026 14:58:16.309350 22123 leader_election.cc:304] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, f6b6590ae6a440d1bcb913208797d2ea; no voters:
I20251026 14:58:16.309430 22232 raft_consensus.cc:2468] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b6590ae6a440d1bcb913208797d2ea in term 3.
I20251026 14:58:16.310014 22471 raft_consensus.cc:2804] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:58:16.310472 22471 raft_consensus.cc:697] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 3 LEADER]: Becoming Leader. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Running, Role: LEADER
I20251026 14:58:16.311192 22471 consensus_queue.cc:237] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:16.318114 22054 catalog_manager.cc:5649] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea reported cstate change: term changed from 2 to 3, leader changed from 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2) to f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1). New cstate: current_term: 3 leader_uuid: "f6b6590ae6a440d1bcb913208797d2ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:16.568264 22306 raft_consensus.cc:1275] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Refusing update from remote peer f6b6590ae6a440d1bcb913208797d2ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:16.569340 22471 consensus_queue.cc:1048] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:16.579840 22306 raft_consensus.cc:1275] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 4 FOLLOWER]: Refusing update from remote peer f6b6590ae6a440d1bcb913208797d2ea: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251026 14:58:16.581594 22232 raft_consensus.cc:1275] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Refusing update from remote peer f6b6590ae6a440d1bcb913208797d2ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:16.583711 22444 consensus_queue.cc:1048] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:58:16.583267 22484 consensus_queue.cc:1048] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:16.594631 22231 raft_consensus.cc:1275] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [term 4 FOLLOWER]: Refusing update from remote peer f6b6590ae6a440d1bcb913208797d2ea: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251026 14:58:16.596315 22484 consensus_queue.cc:1048] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:58:16.631178 22306 raft_consensus.cc:1275] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 3 FOLLOWER]: Refusing update from remote peer f6b6590ae6a440d1bcb913208797d2ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:58:16.632337 22484 consensus_queue.cc:1048] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:16.642860 22232 raft_consensus.cc:1275] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 3 FOLLOWER]: Refusing update from remote peer f6b6590ae6a440d1bcb913208797d2ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:58:16.644099 22471 consensus_queue.cc:1048] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:16.708545 22306 raft_consensus.cc:1275] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 4 FOLLOWER]: Refusing update from remote peer f6b6590ae6a440d1bcb913208797d2ea: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251026 14:58:16.709673 22471 consensus_queue.cc:1048] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:58:16.713693 22232 raft_consensus.cc:1275] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [term 3 FOLLOWER]: Refusing update from remote peer f6b6590ae6a440d1bcb913208797d2ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:58:16.714936 22471 consensus_queue.cc:1048] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:16.716992 22232 raft_consensus.cc:1275] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [term 4 FOLLOWER]: Refusing update from remote peer f6b6590ae6a440d1bcb913208797d2ea: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251026 14:58:16.719556 22444 consensus_queue.cc:1048] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251026 14:58:16.731002 22306 raft_consensus.cc:1275] T c7ad349a3a2c4e6e8235a080aab372b2 P 12bdf853a41f4c14a37d70908b8c2640 [term 3 FOLLOWER]: Refusing update from remote peer f6b6590ae6a440d1bcb913208797d2ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:58:16.732458 22484 consensus_queue.cc:1048] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:19.059257 21512 auto_leader_rebalancer-test.cc:377] The leader distribution is
I20251026 14:58:19.059890 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:58:19.068239 22154 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2cf56c1758324aebad5521cae8b061aa"
dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
mode: GRACEFUL
new_leader_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
from {username='slave'} at 127.0.0.1:53182
I20251026 14:58:19.068847 22154 raft_consensus.cc:606] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 4 LEADER]: Received request to transfer leadership to TS 8b40ffb1f6864b108c15f0b8aefeaced
I20251026 14:58:19.071323 22154 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "304dea4666704f0c81d11c8ed58593cc"
dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
mode: GRACEFUL
new_leader_uuid: "12bdf853a41f4c14a37d70908b8c2640"
from {username='slave'} at 127.0.0.1:53182
I20251026 14:58:19.071870 22154 raft_consensus.cc:606] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Received request to transfer leadership to TS 12bdf853a41f4c14a37d70908b8c2640
I20251026 14:58:19.074049 22154 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4a8c1bc474434dd88d5a7eceeeece989"
dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
mode: GRACEFUL
new_leader_uuid: "12bdf853a41f4c14a37d70908b8c2640"
from {username='slave'} at 127.0.0.1:53182
I20251026 14:58:19.074456 22154 raft_consensus.cc:606] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Received request to transfer leadership to TS 12bdf853a41f4c14a37d70908b8c2640
I20251026 14:58:19.076542 22154 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "da9afe4ffb9547c0b7ff3ed93b58248f"
dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
mode: GRACEFUL
new_leader_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
from {username='slave'} at 127.0.0.1:53182
I20251026 14:58:19.076977 22154 raft_consensus.cc:606] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 4 LEADER]: Received request to transfer leadership to TS 8b40ffb1f6864b108c15f0b8aefeaced
I20251026 14:58:19.077879 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20251026 14:58:19.078332 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:58:19.081579 22497 raft_consensus.cc:993] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea: : Instructing follower 8b40ffb1f6864b108c15f0b8aefeaced to start an election
I20251026 14:58:19.081995 22496 raft_consensus.cc:1081] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 4 LEADER]: Signalling peer 8b40ffb1f6864b108c15f0b8aefeaced to start an election
I20251026 14:58:19.083454 22232 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "da9afe4ffb9547c0b7ff3ed93b58248f"
dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
from {username='slave'} at 127.0.0.1:53740
I20251026 14:58:19.084020 22232 raft_consensus.cc:493] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:58:19.084357 22232 raft_consensus.cc:3060] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:58:19.088671 22232 raft_consensus.cc:515] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:19.090377 22232 leader_election.cc:290] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 5 election: Requested vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:19.091328 22154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da9afe4ffb9547c0b7ff3ed93b58248f" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
I20251026 14:58:19.091328 22305 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da9afe4ffb9547c0b7ff3ed93b58248f" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
I20251026 14:58:19.091997 22154 raft_consensus.cc:3055] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:58:19.092052 22305 raft_consensus.cc:3060] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:58:19.092396 22154 raft_consensus.cc:740] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Running, Role: LEADER
I20251026 14:58:19.093094 22154 consensus_queue.cc:260] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:19.094286 22154 raft_consensus.cc:3060] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:58:19.098564 22305 raft_consensus.cc:2468] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b40ffb1f6864b108c15f0b8aefeaced in term 5.
I20251026 14:58:19.099763 22198 leader_election.cc:304] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, 8b40ffb1f6864b108c15f0b8aefeaced; no voters:
I20251026 14:58:19.100152 22154 raft_consensus.cc:2468] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b40ffb1f6864b108c15f0b8aefeaced in term 5.
I20251026 14:58:19.100365 22478 raft_consensus.cc:2804] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:58:19.101341 22478 raft_consensus.cc:697] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [term 5 LEADER]: Becoming Leader. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Running, Role: LEADER
I20251026 14:58:19.102115 22478 consensus_queue.cc:237] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:19.108628 22053 catalog_manager.cc:5649] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced reported cstate change: term changed from 4 to 5, leader changed from f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1) to 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2). New cstate: current_term: 5 leader_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:19.161509 22497 raft_consensus.cc:993] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea: : Instructing follower 12bdf853a41f4c14a37d70908b8c2640 to start an election
I20251026 14:58:19.161947 22497 raft_consensus.cc:1081] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Signalling peer 12bdf853a41f4c14a37d70908b8c2640 to start an election
I20251026 14:58:19.163297 22305 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "4a8c1bc474434dd88d5a7eceeeece989"
dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
from {username='slave'} at 127.0.0.1:33828
I20251026 14:58:19.163769 22305 raft_consensus.cc:493] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:58:19.164017 22305 raft_consensus.cc:3060] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:19.168471 22305 raft_consensus.cc:515] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:19.170218 22305 leader_election.cc:290] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 3 election: Requested vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737)
I20251026 14:58:19.171190 22154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a8c1bc474434dd88d5a7eceeeece989" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
I20251026 14:58:19.171185 22232 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4a8c1bc474434dd88d5a7eceeeece989" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
I20251026 14:58:19.171713 22154 raft_consensus.cc:3055] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:58:19.171747 22232 raft_consensus.cc:3060] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:19.172031 22154 raft_consensus.cc:740] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Running, Role: LEADER
I20251026 14:58:19.172668 22154 consensus_queue.cc:260] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:19.173578 22154 raft_consensus.cc:3060] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:19.176636 22232 raft_consensus.cc:2468] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 3.
I20251026 14:58:19.177829 22271 leader_election.cc:304] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, 8b40ffb1f6864b108c15f0b8aefeaced; no voters:
I20251026 14:58:19.178073 22154 raft_consensus.cc:2468] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 3.
I20251026 14:58:19.178738 22470 raft_consensus.cc:2804] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:58:19.180063 22470 raft_consensus.cc:697] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 3 LEADER]: Becoming Leader. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Running, Role: LEADER
I20251026 14:58:19.180950 22470 consensus_queue.cc:237] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:19.188302 22053 catalog_manager.cc:5649] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 reported cstate change: term changed from 2 to 3, leader changed from f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1) to 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3). New cstate: current_term: 3 leader_uuid: "12bdf853a41f4c14a37d70908b8c2640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:19.271124 22496 raft_consensus.cc:993] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea: : Instructing follower 8b40ffb1f6864b108c15f0b8aefeaced to start an election
I20251026 14:58:19.271533 22496 raft_consensus.cc:1081] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 4 LEADER]: Signalling peer 8b40ffb1f6864b108c15f0b8aefeaced to start an election
I20251026 14:58:19.272994 22232 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "2cf56c1758324aebad5521cae8b061aa"
dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
from {username='slave'} at 127.0.0.1:53740
I20251026 14:58:19.273458 22232 raft_consensus.cc:493] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:58:19.273698 22232 raft_consensus.cc:3060] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:58:19.278272 22232 raft_consensus.cc:515] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:19.279700 22232 leader_election.cc:290] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 5 election: Requested vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3:40117)
I20251026 14:58:19.280509 22154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2cf56c1758324aebad5521cae8b061aa" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
I20251026 14:58:19.280728 22305 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2cf56c1758324aebad5521cae8b061aa" candidate_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
I20251026 14:58:19.281077 22154 raft_consensus.cc:3055] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:58:19.281162 22305 raft_consensus.cc:3060] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:58:19.281407 22154 raft_consensus.cc:740] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Running, Role: LEADER
I20251026 14:58:19.282047 22154 consensus_queue.cc:260] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:19.282883 22154 raft_consensus.cc:3060] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:58:19.285672 22305 raft_consensus.cc:2468] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b40ffb1f6864b108c15f0b8aefeaced in term 5.
I20251026 14:58:19.286571 22198 leader_election.cc:304] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, 8b40ffb1f6864b108c15f0b8aefeaced; no voters:
I20251026 14:58:19.287184 22478 raft_consensus.cc:2804] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:58:19.287271 22154 raft_consensus.cc:2468] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b40ffb1f6864b108c15f0b8aefeaced in term 5.
I20251026 14:58:19.287716 22478 raft_consensus.cc:697] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [term 5 LEADER]: Becoming Leader. State: Replica: 8b40ffb1f6864b108c15f0b8aefeaced, State: Running, Role: LEADER
I20251026 14:58:19.288569 22478 consensus_queue.cc:237] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:19.296085 22053 catalog_manager.cc:5649] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced reported cstate change: term changed from 4 to 5, leader changed from f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1) to 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2). New cstate: current_term: 5 leader_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:19.462734 22497 raft_consensus.cc:993] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea: : Instructing follower 12bdf853a41f4c14a37d70908b8c2640 to start an election
I20251026 14:58:19.463246 22497 raft_consensus.cc:1081] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Signalling peer 12bdf853a41f4c14a37d70908b8c2640 to start an election
I20251026 14:58:19.464572 22305 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "304dea4666704f0c81d11c8ed58593cc"
dest_uuid: "12bdf853a41f4c14a37d70908b8c2640"
from {username='slave'} at 127.0.0.1:33828
I20251026 14:58:19.464946 22305 raft_consensus.cc:493] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:58:19.465178 22305 raft_consensus.cc:3060] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:19.469561 22305 raft_consensus.cc:515] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:19.470852 22305 leader_election.cc:290] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 3 election: Requested vote from peers f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141), 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737)
I20251026 14:58:19.471758 22154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "304dea4666704f0c81d11c8ed58593cc" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f6b6590ae6a440d1bcb913208797d2ea"
I20251026 14:58:19.471769 22232 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "304dea4666704f0c81d11c8ed58593cc" candidate_uuid: "12bdf853a41f4c14a37d70908b8c2640" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8b40ffb1f6864b108c15f0b8aefeaced"
I20251026 14:58:19.472414 22154 raft_consensus.cc:3055] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:58:19.472535 22232 raft_consensus.cc:3060] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:19.472703 22154 raft_consensus.cc:740] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f6b6590ae6a440d1bcb913208797d2ea, State: Running, Role: LEADER
I20251026 14:58:19.473425 22154 consensus_queue.cc:260] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:19.474278 22154 raft_consensus.cc:3060] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:19.477195 22232 raft_consensus.cc:2468] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 3.
I20251026 14:58:19.478221 22271 leader_election.cc:304] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bdf853a41f4c14a37d70908b8c2640, 8b40ffb1f6864b108c15f0b8aefeaced; no voters:
I20251026 14:58:19.478909 22470 raft_consensus.cc:2804] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:58:19.478955 22154 raft_consensus.cc:2468] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12bdf853a41f4c14a37d70908b8c2640 in term 3.
I20251026 14:58:19.479476 22470 raft_consensus.cc:697] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 [term 3 LEADER]: Becoming Leader. State: Replica: 12bdf853a41f4c14a37d70908b8c2640, State: Running, Role: LEADER
I20251026 14:58:19.480351 22470 consensus_queue.cc:237] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } }
I20251026 14:58:19.487287 22053 catalog_manager.cc:5649] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 reported cstate change: term changed from 2 to 3, leader changed from f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1) to 12bdf853a41f4c14a37d70908b8c2640 (127.21.2.3). New cstate: current_term: 3 leader_uuid: "12bdf853a41f4c14a37d70908b8c2640" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:19.523471 22154 raft_consensus.cc:1275] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 5 FOLLOWER]: Refusing update from remote peer 8b40ffb1f6864b108c15f0b8aefeaced: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251026 14:58:19.524724 22478 consensus_queue.cc:1048] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 14:58:19.533927 22305 raft_consensus.cc:1275] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 5 FOLLOWER]: Refusing update from remote peer 8b40ffb1f6864b108c15f0b8aefeaced: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251026 14:58:19.537420 22478 consensus_queue.cc:1048] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 14:58:19.608830 22232 raft_consensus.cc:1275] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced [term 3 FOLLOWER]: Refusing update from remote peer 12bdf853a41f4c14a37d70908b8c2640: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:58:19.610085 22470 consensus_queue.cc:1048] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:19.618243 22154 raft_consensus.cc:1275] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [term 3 FOLLOWER]: Refusing update from remote peer 12bdf853a41f4c14a37d70908b8c2640: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:58:19.619833 22504 consensus_queue.cc:1048] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:19.732259 22154 raft_consensus.cc:1275] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 5 FOLLOWER]: Refusing update from remote peer 8b40ffb1f6864b108c15f0b8aefeaced: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251026 14:58:19.733780 22498 consensus_queue.cc:1048] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251026 14:58:19.747068 22305 raft_consensus.cc:1275] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 5 FOLLOWER]: Refusing update from remote peer 8b40ffb1f6864b108c15f0b8aefeaced: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251026 14:58:19.749460 22478 consensus_queue.cc:1048] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bdf853a41f4c14a37d70908b8c2640" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251026 14:58:19.988718 22154 raft_consensus.cc:1275] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [term 3 FOLLOWER]: Refusing update from remote peer 12bdf853a41f4c14a37d70908b8c2640: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:58:19.989921 22504 consensus_queue.cc:1048] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b6590ae6a440d1bcb913208797d2ea" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:20.004412 22232 raft_consensus.cc:1275] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 3 FOLLOWER]: Refusing update from remote peer 12bdf853a41f4c14a37d70908b8c2640: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:58:20.005851 22505 consensus_queue.cc:1048] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40ffb1f6864b108c15f0b8aefeaced" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:58:20.079686 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:58:20.086160 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:58:20.086603 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:58:21.087570 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:58:21.093568 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:58:21.094038 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:58:22.094910 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:58:22.100648 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:58:22.101112 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:58:23.102128 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:58:23.107538 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:58:23.108018 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:58:24.109014 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:58:24.114881 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:58:24.115366 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:58:25.116226 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:58:25.121438 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:58:25.121883 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:58:26.122803 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:58:26.128046 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:58:26.128507 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:58:27.129508 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:58:27.135450 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:58:27.135965 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:58:28.136912 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:58:28.142153 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 14:58:28.142558 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:58:29.172200 21512 auto_leader_rebalancer-test.cc:388] The leader distribution is
I20251026 14:58:29.172748 21512 tablet_server.cc:178] TabletServer@127.21.2.1:0 shutting down...
W20251026 14:58:29.195736 22273 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
W20251026 14:58:29.201001 22273 consensus_peers.cc:597] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 -> Peer f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141): Couldn't send request to peer f6b6590ae6a440d1bcb913208797d2ea. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:29.202200 21512 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 14:58:29.202899 21512 tablet_replica.cc:333] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea: stopping tablet replica
I20251026 14:58:29.203533 21512 raft_consensus.cc:2243] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:29.204082 21512 raft_consensus.cc:2272] T 9e0f3fab0c2e40daa0e161d82949044d P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.206061 21512 tablet_replica.cc:333] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea: stopping tablet replica
I20251026 14:58:29.206589 21512 raft_consensus.cc:2243] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 3 LEADER]: Raft consensus shutting down.
I20251026 14:58:29.207381 21512 raft_consensus.cc:2272] T c7ad349a3a2c4e6e8235a080aab372b2 P f6b6590ae6a440d1bcb913208797d2ea [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.209434 21512 tablet_replica.cc:333] T e55b543aef104e7290aa5dae37b421cb P f6b6590ae6a440d1bcb913208797d2ea: stopping tablet replica
I20251026 14:58:29.209992 21512 raft_consensus.cc:2243] T e55b543aef104e7290aa5dae37b421cb P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:29.210453 21512 raft_consensus.cc:2272] T e55b543aef104e7290aa5dae37b421cb P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.212301 21512 tablet_replica.cc:333] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea: stopping tablet replica
I20251026 14:58:29.212807 21512 raft_consensus.cc:2243] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:29.213239 21512 raft_consensus.cc:2272] T 4a8c1bc474434dd88d5a7eceeeece989 P f6b6590ae6a440d1bcb913208797d2ea [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.215090 21512 tablet_replica.cc:333] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea: stopping tablet replica
I20251026 14:58:29.215555 21512 raft_consensus.cc:2243] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Raft consensus shutting down.
I20251026 14:58:29.216325 21512 raft_consensus.cc:2272] T 638b4cb4614f49578bd04fc043b057f3 P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.218163 21512 tablet_replica.cc:333] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea: stopping tablet replica
I20251026 14:58:29.218662 21512 raft_consensus.cc:2243] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:29.219081 21512 raft_consensus.cc:2272] T 304dea4666704f0c81d11c8ed58593cc P f6b6590ae6a440d1bcb913208797d2ea [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.220644 21512 tablet_replica.cc:333] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea: stopping tablet replica
I20251026 14:58:29.221107 21512 raft_consensus.cc:2243] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea [term 3 LEADER]: Raft consensus shutting down.
I20251026 14:58:29.221863 21512 raft_consensus.cc:2272] T f4b102dcc6534e2583371c5b81e2da5b P f6b6590ae6a440d1bcb913208797d2ea [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.223523 21512 tablet_replica.cc:333] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea: stopping tablet replica
I20251026 14:58:29.224001 21512 raft_consensus.cc:2243] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea [term 2 LEADER]: Raft consensus shutting down.
I20251026 14:58:29.224718 21512 raft_consensus.cc:2272] T 9e79d8a8b9394af5b39918f8a0a43b39 P f6b6590ae6a440d1bcb913208797d2ea [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.226382 21512 tablet_replica.cc:333] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea: stopping tablet replica
I20251026 14:58:29.226831 21512 raft_consensus.cc:2243] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:29.227236 21512 raft_consensus.cc:2272] T 2cf56c1758324aebad5521cae8b061aa P f6b6590ae6a440d1bcb913208797d2ea [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.228807 21512 tablet_replica.cc:333] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea: stopping tablet replica
I20251026 14:58:29.229249 21512 raft_consensus.cc:2243] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:29.229667 21512 raft_consensus.cc:2272] T da9afe4ffb9547c0b7ff3ed93b58248f P f6b6590ae6a440d1bcb913208797d2ea [term 5 FOLLOWER]: Raft consensus is shut down!
W20251026 14:58:29.246476 22273 consensus_peers.cc:597] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 -> Peer f6b6590ae6a440d1bcb913208797d2ea (127.21.2.1:36141): Couldn't send request to peer f6b6590ae6a440d1bcb913208797d2ea. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:29.259594 21512 tablet_server.cc:195] TabletServer@127.21.2.1:0 shutdown complete.
I20251026 14:58:29.280802 21512 tablet_server.cc:178] TabletServer@127.21.2.2:0 shutting down...
I20251026 14:58:29.306663 21512 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 14:58:29.307374 21512 tablet_replica.cc:333] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced: stopping tablet replica
I20251026 14:58:29.307986 21512 raft_consensus.cc:2243] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [term 5 LEADER]: Raft consensus shutting down.
I20251026 14:58:29.308936 21512 raft_consensus.cc:2272] T 2cf56c1758324aebad5521cae8b061aa P 8b40ffb1f6864b108c15f0b8aefeaced [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.311165 21512 tablet_replica.cc:333] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced: stopping tablet replica
W20251026 14:58:29.311717 22271 consensus_peers.cc:597] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 -> Peer 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737): Couldn't send request to peer 8b40ffb1f6864b108c15f0b8aefeaced. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:29.311861 21512 raft_consensus.cc:2243] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 LEADER]: Raft consensus shutting down.
I20251026 14:58:29.312716 21512 raft_consensus.cc:2272] T 9e0f3fab0c2e40daa0e161d82949044d P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.314481 21512 tablet_replica.cc:333] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced: stopping tablet replica
I20251026 14:58:29.314980 21512 raft_consensus.cc:2243] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:29.315364 21512 raft_consensus.cc:2272] T c7ad349a3a2c4e6e8235a080aab372b2 P 8b40ffb1f6864b108c15f0b8aefeaced [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.316970 21512 tablet_replica.cc:333] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced: stopping tablet replica
I20251026 14:58:29.317543 21512 raft_consensus.cc:2243] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [term 5 LEADER]: Raft consensus shutting down.
I20251026 14:58:29.318246 21512 raft_consensus.cc:2272] T da9afe4ffb9547c0b7ff3ed93b58248f P 8b40ffb1f6864b108c15f0b8aefeaced [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.319893 21512 tablet_replica.cc:333] T 638b4cb4614f49578bd04fc043b057f3 P 8b40ffb1f6864b108c15f0b8aefeaced: stopping tablet replica
I20251026 14:58:29.320338 21512 raft_consensus.cc:2243] T 638b4cb4614f49578bd04fc043b057f3 P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:29.320713 21512 raft_consensus.cc:2272] T 638b4cb4614f49578bd04fc043b057f3 P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.322342 21512 tablet_replica.cc:333] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced: stopping tablet replica
I20251026 14:58:29.322767 21512 raft_consensus.cc:2243] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:29.323132 21512 raft_consensus.cc:2272] T 304dea4666704f0c81d11c8ed58593cc P 8b40ffb1f6864b108c15f0b8aefeaced [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.324708 21512 tablet_replica.cc:333] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced: stopping tablet replica
I20251026 14:58:29.325213 21512 raft_consensus.cc:2243] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:29.325640 21512 raft_consensus.cc:2272] T f4b102dcc6534e2583371c5b81e2da5b P 8b40ffb1f6864b108c15f0b8aefeaced [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.327307 21512 tablet_replica.cc:333] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced: stopping tablet replica
I20251026 14:58:29.327721 21512 raft_consensus.cc:2243] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:29.328088 21512 raft_consensus.cc:2272] T 9e79d8a8b9394af5b39918f8a0a43b39 P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.329694 21512 tablet_replica.cc:333] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced: stopping tablet replica
I20251026 14:58:29.330101 21512 raft_consensus.cc:2243] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:29.330474 21512 raft_consensus.cc:2272] T e55b543aef104e7290aa5dae37b421cb P 8b40ffb1f6864b108c15f0b8aefeaced [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.332213 21512 tablet_replica.cc:333] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced: stopping tablet replica
I20251026 14:58:29.332660 21512 raft_consensus.cc:2243] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:29.333034 21512 raft_consensus.cc:2272] T 4a8c1bc474434dd88d5a7eceeeece989 P 8b40ffb1f6864b108c15f0b8aefeaced [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.360445 21512 tablet_server.cc:195] TabletServer@127.21.2.2:0 shutdown complete.
I20251026 14:58:29.379653 21512 tablet_server.cc:178] TabletServer@127.21.2.3:0 shutting down...
W20251026 14:58:29.405186 22271 consensus_peers.cc:597] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 -> Peer 8b40ffb1f6864b108c15f0b8aefeaced (127.21.2.2:38737): Couldn't send request to peer 8b40ffb1f6864b108c15f0b8aefeaced. Status: Network error: Client connection negotiation failed: client connection to 127.21.2.2:38737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:29.409775 21512 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 14:58:29.410382 21512 tablet_replica.cc:333] T c7ad349a3a2c4e6e8235a080aab372b2 P 12bdf853a41f4c14a37d70908b8c2640: stopping tablet replica
I20251026 14:58:29.410977 21512 raft_consensus.cc:2243] T c7ad349a3a2c4e6e8235a080aab372b2 P 12bdf853a41f4c14a37d70908b8c2640 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:29.411487 21512 raft_consensus.cc:2272] T c7ad349a3a2c4e6e8235a080aab372b2 P 12bdf853a41f4c14a37d70908b8c2640 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.413416 21512 tablet_replica.cc:333] T 9e0f3fab0c2e40daa0e161d82949044d P 12bdf853a41f4c14a37d70908b8c2640: stopping tablet replica
I20251026 14:58:29.413950 21512 raft_consensus.cc:2243] T 9e0f3fab0c2e40daa0e161d82949044d P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:29.414353 21512 raft_consensus.cc:2272] T 9e0f3fab0c2e40daa0e161d82949044d P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.416044 21512 tablet_replica.cc:333] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640: stopping tablet replica
I20251026 14:58:29.416548 21512 raft_consensus.cc:2243] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:29.416958 21512 raft_consensus.cc:2272] T da9afe4ffb9547c0b7ff3ed93b58248f P 12bdf853a41f4c14a37d70908b8c2640 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.418619 21512 tablet_replica.cc:333] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640: stopping tablet replica
I20251026 14:58:29.419119 21512 raft_consensus.cc:2243] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:29.419538 21512 raft_consensus.cc:2272] T 638b4cb4614f49578bd04fc043b057f3 P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.421175 21512 tablet_replica.cc:333] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640: stopping tablet replica
I20251026 14:58:29.421730 21512 raft_consensus.cc:2243] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 [term 3 LEADER]: Raft consensus shutting down.
I20251026 14:58:29.422452 21512 raft_consensus.cc:2272] T 304dea4666704f0c81d11c8ed58593cc P 12bdf853a41f4c14a37d70908b8c2640 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.424073 21512 tablet_replica.cc:333] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640: stopping tablet replica
I20251026 14:58:29.424542 21512 raft_consensus.cc:2243] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:29.424932 21512 raft_consensus.cc:2272] T 2cf56c1758324aebad5521cae8b061aa P 12bdf853a41f4c14a37d70908b8c2640 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.426502 21512 tablet_replica.cc:333] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640: stopping tablet replica
I20251026 14:58:29.426950 21512 raft_consensus.cc:2243] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [term 2 LEADER]: Raft consensus shutting down.
I20251026 14:58:29.427649 21512 raft_consensus.cc:2272] T e55b543aef104e7290aa5dae37b421cb P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.429314 21512 tablet_replica.cc:333] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640: stopping tablet replica
I20251026 14:58:29.429790 21512 raft_consensus.cc:2243] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 3 LEADER]: Raft consensus shutting down.
I20251026 14:58:29.430490 21512 raft_consensus.cc:2272] T 4a8c1bc474434dd88d5a7eceeeece989 P 12bdf853a41f4c14a37d70908b8c2640 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.432112 21512 tablet_replica.cc:333] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640: stopping tablet replica
I20251026 14:58:29.432544 21512 raft_consensus.cc:2243] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:29.432932 21512 raft_consensus.cc:2272] T f4b102dcc6534e2583371c5b81e2da5b P 12bdf853a41f4c14a37d70908b8c2640 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.434569 21512 tablet_replica.cc:333] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640: stopping tablet replica
I20251026 14:58:29.435006 21512 raft_consensus.cc:2243] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:29.435408 21512 raft_consensus.cc:2272] T 9e79d8a8b9394af5b39918f8a0a43b39 P 12bdf853a41f4c14a37d70908b8c2640 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.462821 21512 tablet_server.cc:195] TabletServer@127.21.2.3:0 shutdown complete.
I20251026 14:58:29.483322 21512 master.cc:561] Master@127.21.2.62:39599 shutting down...
I20251026 14:58:29.506717 21512 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024 [term 1 LEADER]: Raft consensus shutting down.
I20251026 14:58:29.507272 21512 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:29.507545 21512 tablet_replica.cc:333] T 00000000000000000000000000000000 P 477cf6b76d7a46aa946907f6cc2c7024: stopping tablet replica
I20251026 14:58:29.527462 21512 master.cc:583] Master@127.21.2.62:39599 shutdown complete.
12bdf853a41f4c14a37d70908b8c2640 1
8b40ffb1f6864b108c15f0b8aefeaced 4
f6b6590ae6a440d1bcb913208797d2ea 5
12bdf853a41f4c14a37d70908b8c2640 3
8b40ffb1f6864b108c15f0b8aefeaced 4
f6b6590ae6a440d1bcb913208797d2ea 3
12bdf853a41f4c14a37d70908b8c2640 1
8b40ffb1f6864b108c15f0b8aefeaced 1
f6b6590ae6a440d1bcb913208797d2ea 8
12bdf853a41f4c14a37d70908b8c2640 3
8b40ffb1f6864b108c15f0b8aefeaced 3
f6b6590ae6a440d1bcb913208797d2ea 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28823 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20251026 14:58:29.571785 21512 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.21.2.62:38611
I20251026 14:58:29.572870 21512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 14:58:29.578732 22531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 14:58:29.578780 22532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 14:58:29.579262 22534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 14:58:29.580111 21512 server_base.cc:1047] running on GCE node
I20251026 14:58:29.581215 21512 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 14:58:29.581445 21512 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 14:58:29.581620 21512 hybrid_clock.cc:648] HybridClock initialized: now 1761490709581601 us; error 0 us; skew 500 ppm
I20251026 14:58:29.582129 21512 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 14:58:29.584491 21512 webserver.cc:492] Webserver started at http://127.21.2.62:44585/ using document root <none> and password file <none>
I20251026 14:58:29.584952 21512 fs_manager.cc:362] Metadata directory not provided
I20251026 14:58:29.585136 21512 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 14:58:29.585412 21512 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 14:58:29.586587 21512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/master-0-root/instance:
uuid: "a9e76e22adc1466cac02911e2229dc0d"
format_stamp: "Formatted at 2025-10-26 14:58:29 on dist-test-slave-9n4f"
I20251026 14:58:29.591162 21512 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20251026 14:58:29.594815 22539 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 14:58:29.595592 21512 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251026 14:58:29.595904 21512 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/master-0-root
uuid: "a9e76e22adc1466cac02911e2229dc0d"
format_stamp: "Formatted at 2025-10-26 14:58:29 on dist-test-slave-9n4f"
I20251026 14:58:29.596218 21512 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 14:58:29.616102 21512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 14:58:29.617403 21512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 14:58:29.656270 21512 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.62:38611
I20251026 14:58:29.656361 22590 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.2.62:38611 every 8 connection(s)
I20251026 14:58:29.660475 22591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:29.671000 22591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d: Bootstrap starting.
I20251026 14:58:29.675899 22591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:29.680614 22591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d: No bootstrap required, opened a new log
I20251026 14:58:29.682804 22591 raft_consensus.cc:359] T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e76e22adc1466cac02911e2229dc0d" member_type: VOTER }
I20251026 14:58:29.683192 22591 raft_consensus.cc:385] T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:29.683456 22591 raft_consensus.cc:740] T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e76e22adc1466cac02911e2229dc0d, State: Initialized, Role: FOLLOWER
I20251026 14:58:29.684031 22591 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e76e22adc1466cac02911e2229dc0d" member_type: VOTER }
I20251026 14:58:29.684468 22591 raft_consensus.cc:399] T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 14:58:29.684706 22591 raft_consensus.cc:493] T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 14:58:29.684980 22591 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:29.689368 22591 raft_consensus.cc:515] T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e76e22adc1466cac02911e2229dc0d" member_type: VOTER }
I20251026 14:58:29.689893 22591 leader_election.cc:304] T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9e76e22adc1466cac02911e2229dc0d; no voters:
I20251026 14:58:29.691056 22591 leader_election.cc:290] T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 14:58:29.691380 22594 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:29.692755 22594 raft_consensus.cc:697] T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d [term 1 LEADER]: Becoming Leader. State: Replica: a9e76e22adc1466cac02911e2229dc0d, State: Running, Role: LEADER
I20251026 14:58:29.693701 22594 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e76e22adc1466cac02911e2229dc0d" member_type: VOTER }
I20251026 14:58:29.694291 22591 sys_catalog.cc:565] T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d [sys.catalog]: configured and running, proceeding with master startup.
I20251026 14:58:29.700987 22595 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9e76e22adc1466cac02911e2229dc0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e76e22adc1466cac02911e2229dc0d" member_type: VOTER } }
I20251026 14:58:29.701620 22595 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d [sys.catalog]: This master's current role is: LEADER
I20251026 14:58:29.701529 22596 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9e76e22adc1466cac02911e2229dc0d. Latest consensus state: current_term: 1 leader_uuid: "a9e76e22adc1466cac02911e2229dc0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9e76e22adc1466cac02911e2229dc0d" member_type: VOTER } }
I20251026 14:58:29.702065 22596 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d [sys.catalog]: This master's current role is: LEADER
I20251026 14:58:29.703625 22603 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 14:58:29.708596 22603 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 14:58:29.711069 21512 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 14:58:29.718117 22603 catalog_manager.cc:1357] Generated new cluster ID: c6c76253f79a49099bd518e5849eb5eb
I20251026 14:58:29.718465 22603 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 14:58:29.753592 22603 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 14:58:29.755122 22603 catalog_manager.cc:1514] Loading token signing keys...
I20251026 14:58:29.768455 22603 catalog_manager.cc:6022] T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d: Generated new TSK 0
I20251026 14:58:29.769124 22603 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 14:58:29.791692 21512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 14:58:29.799455 22613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 14:58:29.800007 22614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 14:58:29.802940 21512 server_base.cc:1047] running on GCE node
W20251026 14:58:29.802959 22616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 14:58:29.804123 21512 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 14:58:29.804363 21512 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 14:58:29.804513 21512 hybrid_clock.cc:648] HybridClock initialized: now 1761490709804497 us; error 0 us; skew 500 ppm
I20251026 14:58:29.805012 21512 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 14:58:29.807745 21512 webserver.cc:492] Webserver started at http://127.21.2.1:44887/ using document root <none> and password file <none>
I20251026 14:58:29.808269 21512 fs_manager.cc:362] Metadata directory not provided
I20251026 14:58:29.808480 21512 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 14:58:29.808775 21512 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 14:58:29.810154 21512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-0-root/instance:
uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
format_stamp: "Formatted at 2025-10-26 14:58:29 on dist-test-slave-9n4f"
I20251026 14:58:29.814926 21512 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20251026 14:58:29.818400 22621 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 14:58:29.819154 21512 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251026 14:58:29.819458 21512 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-0-root
uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
format_stamp: "Formatted at 2025-10-26 14:58:29 on dist-test-slave-9n4f"
I20251026 14:58:29.819730 21512 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 14:58:29.839679 21512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 14:58:29.841023 21512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 14:58:29.842649 21512 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 14:58:29.845155 21512 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 14:58:29.845412 21512 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20251026 14:58:29.845664 21512 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 14:58:29.845831 21512 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 14:58:29.891216 21512 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.1:46185
I20251026 14:58:29.891330 22683 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.2.1:46185 every 8 connection(s)
I20251026 14:58:29.896286 21512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 14:58:29.904695 22688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 14:58:29.905746 22689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 14:58:29.908550 22691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 14:58:29.910918 22684 heartbeater.cc:344] Connected to a master server at 127.21.2.62:38611
I20251026 14:58:29.911007 21512 server_base.cc:1047] running on GCE node
I20251026 14:58:29.911455 22684 heartbeater.cc:461] Registering TS with master...
I20251026 14:58:29.912202 22684 heartbeater.cc:507] Master 127.21.2.62:38611 requested a full tablet report, sending...
I20251026 14:58:29.912679 21512 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 14:58:29.912899 21512 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 14:58:29.913051 21512 hybrid_clock.cc:648] HybridClock initialized: now 1761490709913035 us; error 0 us; skew 500 ppm
I20251026 14:58:29.913748 21512 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 14:58:29.914647 22556 ts_manager.cc:194] Registered new tserver with Master: df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:58:29.916862 21512 webserver.cc:492] Webserver started at http://127.21.2.2:40809/ using document root <none> and password file <none>
I20251026 14:58:29.917382 22556 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38878
I20251026 14:58:29.917582 21512 fs_manager.cc:362] Metadata directory not provided
I20251026 14:58:29.917847 21512 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 14:58:29.918202 21512 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 14:58:29.919530 21512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-1-root/instance:
uuid: "562bb1e1c8e445b8baecd29c3669658e"
format_stamp: "Formatted at 2025-10-26 14:58:29 on dist-test-slave-9n4f"
I20251026 14:58:29.924857 21512 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20251026 14:58:29.928282 22696 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 14:58:29.929095 21512 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251026 14:58:29.929399 21512 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-1-root
uuid: "562bb1e1c8e445b8baecd29c3669658e"
format_stamp: "Formatted at 2025-10-26 14:58:29 on dist-test-slave-9n4f"
I20251026 14:58:29.929658 21512 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 14:58:29.943225 21512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 14:58:29.944495 21512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 14:58:29.946003 21512 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 14:58:29.948318 21512 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 14:58:29.948493 21512 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 14:58:29.948745 21512 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 14:58:29.948881 21512 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 14:58:29.989768 21512 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.2:46507
I20251026 14:58:29.989851 22758 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.2.2:46507 every 8 connection(s)
I20251026 14:58:29.998184 21512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 14:58:30.004475 22762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 14:58:30.005120 22763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 14:58:30.008121 22765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 14:58:30.008924 21512 server_base.cc:1047] running on GCE node
I20251026 14:58:30.010035 21512 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20251026 14:58:30.010059 22759 heartbeater.cc:344] Connected to a master server at 127.21.2.62:38611
W20251026 14:58:30.010303 21512 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 14:58:30.010512 21512 hybrid_clock.cc:648] HybridClock initialized: now 1761490710010496 us; error 0 us; skew 500 ppm
I20251026 14:58:30.010540 22759 heartbeater.cc:461] Registering TS with master...
I20251026 14:58:30.011076 21512 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 14:58:30.011369 22759 heartbeater.cc:507] Master 127.21.2.62:38611 requested a full tablet report, sending...
I20251026 14:58:30.013698 22556 ts_manager.cc:194] Registered new tserver with Master: 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:30.013818 21512 webserver.cc:492] Webserver started at http://127.21.2.3:35971/ using document root <none> and password file <none>
I20251026 14:58:30.014381 21512 fs_manager.cc:362] Metadata directory not provided
I20251026 14:58:30.014575 21512 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 14:58:30.014825 21512 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 14:58:30.015245 22556 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38880
I20251026 14:58:30.016192 21512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-2-root/instance:
uuid: "0d684445287b450a8f738b6ad4b59821"
format_stamp: "Formatted at 2025-10-26 14:58:30 on dist-test-slave-9n4f"
I20251026 14:58:30.020929 21512 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20251026 14:58:30.025005 22770 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 14:58:30.026131 21512 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251026 14:58:30.026494 21512 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-2-root
uuid: "0d684445287b450a8f738b6ad4b59821"
format_stamp: "Formatted at 2025-10-26 14:58:30 on dist-test-slave-9n4f"
I20251026 14:58:30.026782 21512 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 14:58:30.046959 21512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 14:58:30.048226 21512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 14:58:30.049726 21512 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 14:58:30.052223 21512 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 14:58:30.052420 21512 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 14:58:30.052690 21512 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 14:58:30.052827 21512 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 14:58:30.094123 21512 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.3:39645
I20251026 14:58:30.094429 22832 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.2.3:39645 every 8 connection(s)
I20251026 14:58:30.107962 22833 heartbeater.cc:344] Connected to a master server at 127.21.2.62:38611
I20251026 14:58:30.108343 22833 heartbeater.cc:461] Registering TS with master...
I20251026 14:58:30.109071 22833 heartbeater.cc:507] Master 127.21.2.62:38611 requested a full tablet report, sending...
I20251026 14:58:30.110927 22556 ts_manager.cc:194] Registered new tserver with Master: 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:30.111093 21512 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01371875s
I20251026 14:58:30.112277 22556 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38884
I20251026 14:58:30.113770 21512 test_util.cc:276] Using random seed: 67972565
I20251026 14:58:30.137825 22556 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38890:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 14:58:30.144273 22556 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 14:58:30.353657 22724 tablet_service.cc:1505] Processing CreateTablet for tablet b0734967ed764a3c855f8eab30c49570 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION VALUES < 36398027
I20251026 14:58:30.355015 22793 tablet_service.cc:1505] Processing CreateTablet for tablet 9423e278399742f48ca9ee203d7f92cb (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20251026 14:58:30.356442 22793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9423e278399742f48ca9ee203d7f92cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.357915 22723 tablet_service.cc:1505] Processing CreateTablet for tablet 59b48d4137e3417983741d9cc5f92f42 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20251026 14:58:30.358541 22649 tablet_service.cc:1505] Processing CreateTablet for tablet b0734967ed764a3c855f8eab30c49570 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION VALUES < 36398027
I20251026 14:58:30.360013 22649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0734967ed764a3c855f8eab30c49570. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.360176 22798 tablet_service.cc:1505] Processing CreateTablet for tablet b0734967ed764a3c855f8eab30c49570 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION VALUES < 36398027
I20251026 14:58:30.362824 22798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0734967ed764a3c855f8eab30c49570. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.362557 22791 tablet_service.cc:1505] Processing CreateTablet for tablet 524d8339d5244ca98e86f0ff2fd02b67 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20251026 14:58:30.363794 22719 tablet_service.cc:1505] Processing CreateTablet for tablet 9423e278399742f48ca9ee203d7f92cb (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20251026 14:58:30.366339 22795 tablet_service.cc:1505] Processing CreateTablet for tablet 469842eacb94455aaff74df534ff0bf1 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20251026 14:58:30.366600 22647 tablet_service.cc:1505] Processing CreateTablet for tablet fbb846c2db3b4f44b2245a7493b36c7c (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20251026 14:58:30.367789 22795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 469842eacb94455aaff74df534ff0bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.367990 22647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbb846c2db3b4f44b2245a7493b36c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.370565 22720 tablet_service.cc:1505] Processing CreateTablet for tablet 6a1f6f52f31d4a9e98b2c86f2358ea8b (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20251026 14:58:30.371842 22644 tablet_service.cc:1505] Processing CreateTablet for tablet 9423e278399742f48ca9ee203d7f92cb (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20251026 14:58:30.373076 22796 tablet_service.cc:1505] Processing CreateTablet for tablet fbb846c2db3b4f44b2245a7493b36c7c (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20251026 14:58:30.374517 22792 tablet_service.cc:1505] Processing CreateTablet for tablet f688f64ce4e84d419d88768435bf773f (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20251026 14:58:30.375730 22721 tablet_service.cc:1505] Processing CreateTablet for tablet 469842eacb94455aaff74df534ff0bf1 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20251026 14:58:30.374519 22796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbb846c2db3b4f44b2245a7493b36c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.376888 22643 tablet_service.cc:1505] Processing CreateTablet for tablet f688f64ce4e84d419d88768435bf773f (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20251026 14:58:30.378948 22797 tablet_service.cc:1505] Processing CreateTablet for tablet 59b48d4137e3417983741d9cc5f92f42 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20251026 14:58:30.370575 22646 tablet_service.cc:1505] Processing CreateTablet for tablet 469842eacb94455aaff74df534ff0bf1 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20251026 14:58:30.379966 22792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f688f64ce4e84d419d88768435bf773f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.377899 22644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9423e278399742f48ca9ee203d7f92cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.362587 22722 tablet_service.cc:1505] Processing CreateTablet for tablet fbb846c2db3b4f44b2245a7493b36c7c (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20251026 14:58:30.381314 22643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f688f64ce4e84d419d88768435bf773f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.365032 22648 tablet_service.cc:1505] Processing CreateTablet for tablet 59b48d4137e3417983741d9cc5f92f42 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20251026 14:58:30.381230 22642 tablet_service.cc:1505] Processing CreateTablet for tablet 524d8339d5244ca98e86f0ff2fd02b67 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20251026 14:58:30.382485 22648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59b48d4137e3417983741d9cc5f92f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.362587 22794 tablet_service.cc:1505] Processing CreateTablet for tablet 6a1f6f52f31d4a9e98b2c86f2358ea8b (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20251026 14:58:30.384873 22642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 524d8339d5244ca98e86f0ff2fd02b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.356815 22645 tablet_service.cc:1505] Processing CreateTablet for tablet 6a1f6f52f31d4a9e98b2c86f2358ea8b (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20251026 14:58:30.386372 22645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a1f6f52f31d4a9e98b2c86f2358ea8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.396978 22641 tablet_service.cc:1505] Processing CreateTablet for tablet c9339fcac3bb4aa1949bdfb69eeedc2d (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20251026 14:58:30.399121 22641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9339fcac3bb4aa1949bdfb69eeedc2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.400393 22646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 469842eacb94455aaff74df534ff0bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.401520 22717 tablet_service.cc:1505] Processing CreateTablet for tablet 524d8339d5244ca98e86f0ff2fd02b67 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20251026 14:58:30.403600 22718 tablet_service.cc:1505] Processing CreateTablet for tablet f688f64ce4e84d419d88768435bf773f (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20251026 14:58:30.355499 22724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0734967ed764a3c855f8eab30c49570. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.409598 22791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 524d8339d5244ca98e86f0ff2fd02b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.409554 22716 tablet_service.cc:1505] Processing CreateTablet for tablet c9339fcac3bb4aa1949bdfb69eeedc2d (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20251026 14:58:30.414273 22794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a1f6f52f31d4a9e98b2c86f2358ea8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.414944 22797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59b48d4137e3417983741d9cc5f92f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.416488 22790 tablet_service.cc:1505] Processing CreateTablet for tablet c9339fcac3bb4aa1949bdfb69eeedc2d (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20251026 14:58:30.424671 22640 tablet_service.cc:1505] Processing CreateTablet for tablet 6b172938cfc843318b60a7fc78190d2c (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20251026 14:58:30.426270 22640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b172938cfc843318b60a7fc78190d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.430313 22790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9339fcac3bb4aa1949bdfb69eeedc2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.441613 22721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 469842eacb94455aaff74df534ff0bf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.442333 22723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59b48d4137e3417983741d9cc5f92f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.443537 22716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9339fcac3bb4aa1949bdfb69eeedc2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.444965 22717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 524d8339d5244ca98e86f0ff2fd02b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.446358 22718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f688f64ce4e84d419d88768435bf773f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.453781 22719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9423e278399742f48ca9ee203d7f92cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.461688 22720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a1f6f52f31d4a9e98b2c86f2358ea8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.472455 22715 tablet_service.cc:1505] Processing CreateTablet for tablet 6b172938cfc843318b60a7fc78190d2c (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20251026 14:58:30.473654 22722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbb846c2db3b4f44b2245a7493b36c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.474366 22715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b172938cfc843318b60a7fc78190d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.488675 22798 tablet_service.cc:1505] Processing CreateTablet for tablet 6b172938cfc843318b60a7fc78190d2c (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20251026 14:58:30.489668 22646 tablet_service.cc:1505] Processing CreateTablet for tablet 2e6d661a639541b3a40bf3c1fc6c0a88 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20251026 14:58:30.490142 22798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b172938cfc843318b60a7fc78190d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.490971 22646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e6d661a639541b3a40bf3c1fc6c0a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.501497 22794 tablet_service.cc:1505] Processing CreateTablet for tablet 2e6d661a639541b3a40bf3c1fc6c0a88 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20251026 14:58:30.502458 22647 tablet_service.cc:1505] Processing CreateTablet for tablet 7de11d91eebd4cd0b3183e4c044d4004 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20251026 14:58:30.503790 22647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7de11d91eebd4cd0b3183e4c044d4004. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.502866 22794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e6d661a639541b3a40bf3c1fc6c0a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.523156 22793 tablet_service.cc:1505] Processing CreateTablet for tablet 7de11d91eebd4cd0b3183e4c044d4004 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20251026 14:58:30.529786 22793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7de11d91eebd4cd0b3183e4c044d4004. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.548281 22849 tablet_bootstrap.cc:492] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:30.550092 22848 tablet_bootstrap.cc:492] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:30.552822 22723 tablet_service.cc:1505] Processing CreateTablet for tablet 7de11d91eebd4cd0b3183e4c044d4004 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20251026 14:58:30.553081 22719 tablet_service.cc:1505] Processing CreateTablet for tablet 2e6d661a639541b3a40bf3c1fc6c0a88 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20251026 14:58:30.570910 22643 tablet_service.cc:1505] Processing CreateTablet for tablet 45fb267b6944430e8237d9baad843789 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20251026 14:58:30.556632 22791 tablet_service.cc:1505] Processing CreateTablet for tablet 2efb26bccebf43b5b8a85f51fd977714 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20251026 14:58:30.572614 22643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45fb267b6944430e8237d9baad843789. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.572751 22791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2efb26bccebf43b5b8a85f51fd977714. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.570653 22715 tablet_service.cc:1505] Processing CreateTablet for tablet 2efb26bccebf43b5b8a85f51fd977714 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20251026 14:58:30.583635 22715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2efb26bccebf43b5b8a85f51fd977714. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.585006 22848 tablet_bootstrap.cc:654] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:30.586361 22640 tablet_service.cc:1505] Processing CreateTablet for tablet 2efb26bccebf43b5b8a85f51fd977714 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20251026 14:58:30.587687 22640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2efb26bccebf43b5b8a85f51fd977714. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.589062 22719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e6d661a639541b3a40bf3c1fc6c0a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.589797 22723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7de11d91eebd4cd0b3183e4c044d4004. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.592751 22716 tablet_service.cc:1505] Processing CreateTablet for tablet 45fb267b6944430e8237d9baad843789 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20251026 14:58:30.598644 22716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45fb267b6944430e8237d9baad843789. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.598928 22849 tablet_bootstrap.cc:654] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:30.620800 22850 tablet_bootstrap.cc:492] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:30.647943 22645 tablet_service.cc:1505] Processing CreateTablet for tablet a35eb71706aa4f97bf6c64430916aef5 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20251026 14:58:30.649406 22645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a35eb71706aa4f97bf6c64430916aef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.656366 22794 tablet_service.cc:1505] Processing CreateTablet for tablet 45fb267b6944430e8237d9baad843789 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20251026 14:58:30.658066 22850 tablet_bootstrap.cc:654] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:30.658282 22794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45fb267b6944430e8237d9baad843789. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.670684 22796 tablet_service.cc:1505] Processing CreateTablet for tablet a35eb71706aa4f97bf6c64430916aef5 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20251026 14:58:30.673349 22796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a35eb71706aa4f97bf6c64430916aef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.675489 22722 tablet_service.cc:1505] Processing CreateTablet for tablet ba8784cbec044fb7b5b098961f0f94b2 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20251026 14:58:30.676770 22722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba8784cbec044fb7b5b098961f0f94b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.676712 22798 tablet_service.cc:1505] Processing CreateTablet for tablet ba8784cbec044fb7b5b098961f0f94b2 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20251026 14:58:30.679625 22798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba8784cbec044fb7b5b098961f0f94b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.683207 22648 tablet_service.cc:1505] Processing CreateTablet for tablet 5c9a78d6a2b442eabd005c8378b68733 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20251026 14:58:30.684620 22648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c9a78d6a2b442eabd005c8378b68733. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.699121 22849 tablet_bootstrap.cc:492] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:30.699692 22849 ts_tablet_manager.cc:1403] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.152s user 0.011s sys 0.008s
I20251026 14:58:30.702385 22721 tablet_service.cc:1505] Processing CreateTablet for tablet a35eb71706aa4f97bf6c64430916aef5 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20251026 14:58:30.704077 22721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a35eb71706aa4f97bf6c64430916aef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.704089 22717 tablet_service.cc:1505] Processing CreateTablet for tablet 5c9a78d6a2b442eabd005c8378b68733 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20251026 14:58:30.707885 22792 tablet_service.cc:1505] Processing CreateTablet for tablet 5c9a78d6a2b442eabd005c8378b68733 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20251026 14:58:30.708735 22717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c9a78d6a2b442eabd005c8378b68733. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.709196 22792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c9a78d6a2b442eabd005c8378b68733. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.709794 22643 tablet_service.cc:1505] Processing CreateTablet for tablet 6d267e50950c4ef2b95c745e7103cafb (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20251026 14:58:30.707885 22649 tablet_service.cc:1505] Processing CreateTablet for tablet ba8784cbec044fb7b5b098961f0f94b2 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20251026 14:58:30.711100 22643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d267e50950c4ef2b95c745e7103cafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.711557 22848 tablet_bootstrap.cc:492] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:30.711809 22649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba8784cbec044fb7b5b098961f0f94b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.712044 22848 ts_tablet_manager.cc:1403] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.162s user 0.010s sys 0.010s
W20251026 14:58:30.706851 22604 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251026 14:58:30.722930 22849 raft_consensus.cc:359] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:30.724035 22849 raft_consensus.cc:385] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:30.724363 22849 raft_consensus.cc:740] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:30.731667 22848 raft_consensus.cc:359] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:30.732472 22848 raft_consensus.cc:385] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:30.732769 22848 raft_consensus.cc:740] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:30.733541 22848 consensus_queue.cc:260] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:30.725135 22849 consensus_queue.cc:260] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:30.738916 22717 tablet_service.cc:1505] Processing CreateTablet for tablet 6d267e50950c4ef2b95c745e7103cafb (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20251026 14:58:30.740372 22717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d267e50950c4ef2b95c745e7103cafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.745162 22647 tablet_service.cc:1505] Processing CreateTablet for tablet 11ae57d90c614ce2993629f95b777120 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20251026 14:58:30.746608 22647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11ae57d90c614ce2993629f95b777120. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.751230 22798 tablet_service.cc:1505] Processing CreateTablet for tablet 6d267e50950c4ef2b95c745e7103cafb (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20251026 14:58:30.752786 22798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d267e50950c4ef2b95c745e7103cafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.753187 22715 tablet_service.cc:1505] Processing CreateTablet for tablet 11ae57d90c614ce2993629f95b777120 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20251026 14:58:30.755035 22715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11ae57d90c614ce2993629f95b777120. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.762912 22850 tablet_bootstrap.cc:492] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:30.763463 22850 ts_tablet_manager.cc:1403] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.143s user 0.026s sys 0.020s
I20251026 14:58:30.770470 22684 heartbeater.cc:499] Master 127.21.2.62:38611 was elected leader, sending a full tablet report...
I20251026 14:58:30.770467 22848 ts_tablet_manager.cc:1434] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.058s user 0.018s sys 0.005s
I20251026 14:58:30.771416 22794 tablet_service.cc:1505] Processing CreateTablet for tablet 11ae57d90c614ce2993629f95b777120 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20251026 14:58:30.771909 22848 tablet_bootstrap.cc:492] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:30.772871 22794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11ae57d90c614ce2993629f95b777120. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.773595 22850 raft_consensus.cc:359] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:30.774427 22850 raft_consensus.cc:385] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:30.774763 22850 raft_consensus.cc:740] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:30.775590 22850 consensus_queue.cc:260] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:30.778842 22833 heartbeater.cc:499] Master 127.21.2.62:38611 was elected leader, sending a full tablet report...
I20251026 14:58:30.780787 22716 tablet_service.cc:1505] Processing CreateTablet for tablet 4e7ddd2458b64111839b7740daabf522 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20251026 14:58:30.782243 22716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e7ddd2458b64111839b7740daabf522. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.782732 22640 tablet_service.cc:1505] Processing CreateTablet for tablet 4e7ddd2458b64111839b7740daabf522 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20251026 14:58:30.784169 22640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e7ddd2458b64111839b7740daabf522. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.804116 22848 tablet_bootstrap.cc:654] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:30.802394 22796 tablet_service.cc:1505] Processing CreateTablet for tablet 4e7ddd2458b64111839b7740daabf522 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20251026 14:58:30.806048 22649 tablet_service.cc:1505] Processing CreateTablet for tablet 4dea3b45142b41baacfa105498c9e04e (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20251026 14:58:30.815457 22649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dea3b45142b41baacfa105498c9e04e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.817965 22849 ts_tablet_manager.cc:1434] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.118s user 0.009s sys 0.016s
I20251026 14:58:30.819051 22849 tablet_bootstrap.cc:492] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:30.818792 22791 tablet_service.cc:1505] Processing CreateTablet for tablet 4dea3b45142b41baacfa105498c9e04e (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20251026 14:58:30.820123 22791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dea3b45142b41baacfa105498c9e04e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.826879 22643 tablet_service.cc:1505] Processing CreateTablet for tablet 7f59b567deee4fd6bc1428954f8af0ad (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20251026 14:58:30.828811 22643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f59b567deee4fd6bc1428954f8af0ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.830281 22796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e7ddd2458b64111839b7740daabf522. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.839784 22798 tablet_service.cc:1505] Processing CreateTablet for tablet 7f59b567deee4fd6bc1428954f8af0ad (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20251026 14:58:30.841326 22798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f59b567deee4fd6bc1428954f8af0ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.846704 22850 ts_tablet_manager.cc:1434] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.083s user 0.008s sys 0.002s
I20251026 14:58:30.847525 22759 heartbeater.cc:499] Master 127.21.2.62:38611 was elected leader, sending a full tablet report...
I20251026 14:58:30.856387 22649 tablet_service.cc:1505] Processing CreateTablet for tablet 3620632bd772435f97f799be96d493b8 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20251026 14:58:30.859751 22648 tablet_service.cc:1505] Processing CreateTablet for tablet 3ea63385804c40689502aefb2b695e78 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20251026 14:58:30.861929 22716 tablet_service.cc:1505] Processing CreateTablet for tablet 4dea3b45142b41baacfa105498c9e04e (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20251026 14:58:30.862406 22719 tablet_service.cc:1505] Processing CreateTablet for tablet 3ea63385804c40689502aefb2b695e78 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20251026 14:58:30.863296 22716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dea3b45142b41baacfa105498c9e04e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.865334 22724 tablet_service.cc:1505] Processing CreateTablet for tablet 741e08995c924c8e9ea37c02c41a5bc7 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20251026 14:58:30.865447 22722 tablet_service.cc:1505] Processing CreateTablet for tablet 7f59b567deee4fd6bc1428954f8af0ad (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20251026 14:58:30.866287 22723 tablet_service.cc:1505] Processing CreateTablet for tablet 3620632bd772435f97f799be96d493b8 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20251026 14:58:30.866775 22724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 741e08995c924c8e9ea37c02c41a5bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.867611 22723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3620632bd772435f97f799be96d493b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.871690 22718 tablet_service.cc:1505] Processing CreateTablet for tablet 1badbeef1a1c4427bbfa1b762c7d0739 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20251026 14:58:30.871984 22646 tablet_service.cc:1505] Processing CreateTablet for tablet 1badbeef1a1c4427bbfa1b762c7d0739 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20251026 14:58:30.873175 22718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1badbeef1a1c4427bbfa1b762c7d0739. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.874707 22856 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:30.874817 22790 tablet_service.cc:1505] Processing CreateTablet for tablet 3ea63385804c40689502aefb2b695e78 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20251026 14:58:30.875345 22856 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:30.875770 22850 tablet_bootstrap.cc:492] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:30.876250 22790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ea63385804c40689502aefb2b695e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.875876 22793 tablet_service.cc:1505] Processing CreateTablet for tablet 741e08995c924c8e9ea37c02c41a5bc7 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20251026 14:58:30.871904 22645 tablet_service.cc:1505] Processing CreateTablet for tablet 741e08995c924c8e9ea37c02c41a5bc7 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20251026 14:58:30.877365 22793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 741e08995c924c8e9ea37c02c41a5bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.887265 22850 tablet_bootstrap.cc:654] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:30.887529 22849 tablet_bootstrap.cc:654] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:30.894384 22792 tablet_service.cc:1505] Processing CreateTablet for tablet 3620632bd772435f97f799be96d493b8 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20251026 14:58:30.895524 22649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3620632bd772435f97f799be96d493b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.908085 22792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3620632bd772435f97f799be96d493b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.912168 22856 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:30.922159 22719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ea63385804c40689502aefb2b695e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.925724 22722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f59b567deee4fd6bc1428954f8af0ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.946916 22648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ea63385804c40689502aefb2b695e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.974203 22645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 741e08995c924c8e9ea37c02c41a5bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.985797 22646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1badbeef1a1c4427bbfa1b762c7d0739. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:30.996541 22848 tablet_bootstrap.cc:492] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:30.997236 22848 ts_tablet_manager.cc:1403] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.226s user 0.015s sys 0.042s
I20251026 14:58:31.015224 22849 tablet_bootstrap.cc:492] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:31.019014 22849 ts_tablet_manager.cc:1403] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.200s user 0.016s sys 0.008s
I20251026 14:58:31.021170 22850 tablet_bootstrap.cc:492] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:31.021716 22850 ts_tablet_manager.cc:1403] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.146s user 0.014s sys 0.008s
I20251026 14:58:31.023173 22849 raft_consensus.cc:359] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:58:31.024343 22849 raft_consensus.cc:385] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:31.024468 22850 raft_consensus.cc:359] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.024857 22849 raft_consensus.cc:740] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:31.025857 22849 consensus_queue.cc:260] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:58:31.028304 22849 ts_tablet_manager.cc:1434] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20251026 14:58:31.028398 22717 tablet_service.cc:1505] Processing CreateTablet for tablet 37bde08107a44bac947149e44937c2e6 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20251026 14:58:31.028973 22718 tablet_service.cc:1505] Processing CreateTablet for tablet 3b18a8235bfa4d2cb35c7976ce46e14c (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20251026 14:58:31.029744 22849 tablet_bootstrap.cc:492] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:31.031180 22715 tablet_service.cc:1505] Processing CreateTablet for tablet 02799da19b36403499a63101b8418ffe (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20251026 14:58:31.029775 22717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37bde08107a44bac947149e44937c2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.032527 22715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02799da19b36403499a63101b8418ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.033293 22721 tablet_service.cc:1505] Processing CreateTablet for tablet 5b0536aab3b541a2a846d187e111c09c (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20251026 14:58:31.034544 22721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b0536aab3b541a2a846d187e111c09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.036988 22849 tablet_bootstrap.cc:654] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:31.042583 22796 tablet_service.cc:1505] Processing CreateTablet for tablet 1badbeef1a1c4427bbfa1b762c7d0739 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20251026 14:58:31.044010 22796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1badbeef1a1c4427bbfa1b762c7d0739. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.044214 22794 tablet_service.cc:1505] Processing CreateTablet for tablet 02799da19b36403499a63101b8418ffe (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20251026 14:58:31.045490 22794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02799da19b36403499a63101b8418ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.046295 22792 tablet_service.cc:1505] Processing CreateTablet for tablet 3b18a8235bfa4d2cb35c7976ce46e14c (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20251026 14:58:31.047574 22792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b18a8235bfa4d2cb35c7976ce46e14c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.050300 22798 tablet_service.cc:1505] Processing CreateTablet for tablet 37bde08107a44bac947149e44937c2e6 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20251026 14:58:31.051602 22798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37bde08107a44bac947149e44937c2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.052026 22643 tablet_service.cc:1505] Processing CreateTablet for tablet 37bde08107a44bac947149e44937c2e6 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20251026 14:58:31.053850 22647 tablet_service.cc:1505] Processing CreateTablet for tablet 02799da19b36403499a63101b8418ffe (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20251026 14:58:31.054319 22640 tablet_service.cc:1505] Processing CreateTablet for tablet 3b18a8235bfa4d2cb35c7976ce46e14c (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20251026 14:58:31.056005 22848 raft_consensus.cc:359] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.056211 22641 tablet_service.cc:1505] Processing CreateTablet for tablet 5b0536aab3b541a2a846d187e111c09c (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20251026 14:58:31.056826 22848 raft_consensus.cc:385] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:31.057613 22718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b18a8235bfa4d2cb35c7976ce46e14c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.058043 22642 tablet_service.cc:1505] Processing CreateTablet for tablet 3103efcd5601429a95eeb7a270f6fab1 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20251026 14:58:31.058153 22720 tablet_service.cc:1505] Processing CreateTablet for tablet 3103efcd5601429a95eeb7a270f6fab1 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20251026 14:58:31.058043 22791 tablet_service.cc:1505] Processing CreateTablet for tablet 5b0536aab3b541a2a846d187e111c09c (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20251026 14:58:31.059360 22643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37bde08107a44bac947149e44937c2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.059561 22720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3103efcd5601429a95eeb7a270f6fab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.059646 22791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b0536aab3b541a2a846d187e111c09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.057154 22848 raft_consensus.cc:740] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:31.025218 22850 raft_consensus.cc:385] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:31.068645 22854 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:31.069427 22854 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.072686 22854 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:31.075232 22850 raft_consensus.cc:740] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:31.075357 22797 tablet_service.cc:1505] Processing CreateTablet for tablet 3103efcd5601429a95eeb7a270f6fab1 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20251026 14:58:31.076138 22850 consensus_queue.cc:260] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.076712 22797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3103efcd5601429a95eeb7a270f6fab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.094136 22848 consensus_queue.cc:260] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.096755 22848 ts_tablet_manager.cc:1434] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.086s user 0.007s sys 0.000s
I20251026 14:58:31.100430 22856 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:31.100965 22856 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.103040 22856 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:31.104602 22644 tablet_service.cc:1505] Processing CreateTablet for tablet ef8c3fc001e34801b3f872531b651e4d (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20251026 14:58:31.124555 22855 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:31.125142 22855 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.129597 22855 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:31.132786 22795 tablet_service.cc:1505] Processing CreateTablet for tablet ef8c3fc001e34801b3f872531b651e4d (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20251026 14:58:31.146660 22795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef8c3fc001e34801b3f872531b651e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.152900 22719 tablet_service.cc:1505] Processing CreateTablet for tablet ef8c3fc001e34801b3f872531b651e4d (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20251026 14:58:31.154412 22719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef8c3fc001e34801b3f872531b651e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.165661 22647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02799da19b36403499a63101b8418ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.166142 22848 tablet_bootstrap.cc:492] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:31.169869 22644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef8c3fc001e34801b3f872531b651e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.173871 22642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3103efcd5601429a95eeb7a270f6fab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.177642 22640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b18a8235bfa4d2cb35c7976ce46e14c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.185920 22850 ts_tablet_manager.cc:1434] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.164s user 0.008s sys 0.000s
I20251026 14:58:31.186944 22850 tablet_bootstrap.cc:492] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:31.188498 22641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b0536aab3b541a2a846d187e111c09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.202589 22798 tablet_service.cc:1505] Processing CreateTablet for tablet 9ab2875267d54ec89df26641d40fbb40 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20251026 14:58:31.207607 22645 tablet_service.cc:1505] Processing CreateTablet for tablet 4129daa4fb70408193f07b8e6d25effe (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20251026 14:58:31.209842 22645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4129daa4fb70408193f07b8e6d25effe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.211340 22798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ab2875267d54ec89df26641d40fbb40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.217259 22850 tablet_bootstrap.cc:654] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:31.223778 22648 tablet_service.cc:1505] Processing CreateTablet for tablet 9ab2875267d54ec89df26641d40fbb40 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20251026 14:58:31.229557 22648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ab2875267d54ec89df26641d40fbb40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.230716 22855 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:31.236790 22855 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:58:31.238282 22797 tablet_service.cc:1505] Processing CreateTablet for tablet 4129daa4fb70408193f07b8e6d25effe (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20251026 14:58:31.238754 22855 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:58:31.239655 22797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4129daa4fb70408193f07b8e6d25effe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.248450 22850 tablet_bootstrap.cc:492] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:31.248797 22848 tablet_bootstrap.cc:654] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:31.249404 22850 ts_tablet_manager.cc:1403] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.063s user 0.011s sys 0.013s
I20251026 14:58:31.254383 22724 tablet_service.cc:1505] Processing CreateTablet for tablet 4129daa4fb70408193f07b8e6d25effe (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20251026 14:58:31.256233 22724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4129daa4fb70408193f07b8e6d25effe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.277989 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:31.280627 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:31.281371 22659 raft_consensus.cc:2468] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:31.281623 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:31.282338 22734 raft_consensus.cc:2468] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 0.
I20251026 14:58:31.283716 22849 tablet_bootstrap.cc:492] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:31.284222 22849 ts_tablet_manager.cc:1403] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.255s user 0.010s sys 0.012s
I20251026 14:58:31.284981 22624 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:31.285810 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:31.287621 22622 leader_election.cc:343] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:31.290594 22849 raft_consensus.cc:359] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.290966 22854 raft_consensus.cc:2804] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:31.291564 22854 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:31.291900 22854 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:31.300132 22854 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.301411 22700 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:31.301903 22849 raft_consensus.cc:385] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:31.306207 22649 tablet_service.cc:1505] Processing CreateTablet for tablet e2b59eefda3b4f2d93934f02c4981c62 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20251026 14:58:31.316715 22716 tablet_service.cc:1505] Processing CreateTablet for tablet 9ab2875267d54ec89df26641d40fbb40 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20251026 14:58:31.324041 22649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2b59eefda3b4f2d93934f02c4981c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.324734 22850 raft_consensus.cc:359] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.325796 22850 raft_consensus.cc:385] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:31.326197 22850 raft_consensus.cc:740] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:31.327335 22850 consensus_queue.cc:260] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.329066 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:58:31.331503 22697 leader_election.cc:343] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:31.320322 22849 raft_consensus.cc:740] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:31.332788 22849 consensus_queue.cc:260] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.333081 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:31.335367 22849 ts_tablet_manager.cc:1434] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.051s user 0.006s sys 0.002s
I20251026 14:58:31.336179 22849 tablet_bootstrap.cc:492] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:31.350059 22716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ab2875267d54ec89df26641d40fbb40. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 14:58:31.352388 22697 leader_election.cc:343] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:31.358829 22855 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:31.359809 22855 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.361472 22849 tablet_bootstrap.cc:654] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:31.362972 22855 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:31.367600 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:31.368348 22658 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:31.369632 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:31.372236 22856 raft_consensus.cc:2804] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:31.372596 22856 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:31.372900 22856 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:31.373097 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:58:31.374234 22622 leader_election.cc:343] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:31.377225 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:31.370452 22854 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:31.386159 22774 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:58:31.387223 22774 leader_election.cc:343] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:31.389304 22850 ts_tablet_manager.cc:1434] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.139s user 0.003s sys 0.005s
I20251026 14:58:31.389359 22855 raft_consensus.cc:2804] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:31.389120 22644 tablet_service.cc:1505] Processing CreateTablet for tablet 82201a13e5354c8ea3b41436df180125 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20251026 14:58:31.389803 22855 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:31.390197 22855 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:31.390470 22850 tablet_bootstrap.cc:492] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:31.390651 22644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82201a13e5354c8ea3b41436df180125. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.394811 22791 tablet_service.cc:1505] Processing CreateTablet for tablet 82201a13e5354c8ea3b41436df180125 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20251026 14:58:31.389127 22796 tablet_service.cc:1505] Processing CreateTablet for tablet 5a0e9c0787aa4053afa2791a86bb263a (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20251026 14:58:31.396742 22646 tablet_service.cc:1505] Processing CreateTablet for tablet 5a0e9c0787aa4053afa2791a86bb263a (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
W20251026 14:58:31.381848 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 80677us
W20251026 14:58:31.401687 22700 leader_election.cc:343] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:31.402108 22700 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:58:31.403226 22774 leader_election.cc:343] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:31.398097 22646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a0e9c0787aa4053afa2791a86bb263a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.400225 22791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82201a13e5354c8ea3b41436df180125. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.405647 22796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a0e9c0787aa4053afa2791a86bb263a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.371122 22798 tablet_service.cc:1505] Processing CreateTablet for tablet e2b59eefda3b4f2d93934f02c4981c62 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20251026 14:58:31.414727 22798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2b59eefda3b4f2d93934f02c4981c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.418660 22855 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.421685 22848 tablet_bootstrap.cc:492] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:31.422274 22850 tablet_bootstrap.cc:654] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:31.437825 22848 ts_tablet_manager.cc:1403] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.272s user 0.031s sys 0.053s
I20251026 14:58:31.438694 22795 tablet_service.cc:1505] Processing CreateTablet for tablet dea6ba9f8ecc4124bfedb6febafde72b (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20251026 14:58:31.423707 22856 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.429355 22647 tablet_service.cc:1505] Processing CreateTablet for tablet ea1e7da57b65433b9e01cad8c4adeb75 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20251026 14:58:31.440052 22795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dea6ba9f8ecc4124bfedb6febafde72b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.441180 22647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea1e7da57b65433b9e01cad8c4adeb75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.441833 22856 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:31.442409 22856 raft_consensus.cc:2749] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:31.443643 22855 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:31.447067 22856 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:31.447558 22856 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.440059 22848 raft_consensus.cc:359] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.449920 22848 raft_consensus.cc:385] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:31.450335 22848 raft_consensus.cc:740] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:31.450838 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:31.451689 22807 raft_consensus.cc:2393] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 1: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:31.436952 22849 tablet_bootstrap.cc:492] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:31.444126 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:31.452355 22659 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:31.453047 22856 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:31.446556 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:58:31.458526 22697 leader_election.cc:343] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:31.434999 22798 tablet_service.cc:1505] Processing CreateTablet for tablet ea1e7da57b65433b9e01cad8c4adeb75 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20251026 14:58:31.460311 22642 tablet_service.cc:1505] Processing CreateTablet for tablet 9440e0456e0648a5af21108f5fd7cfba (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20251026 14:58:31.451351 22848 consensus_queue.cc:260] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.461855 22642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9440e0456e0648a5af21108f5fd7cfba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.465680 22848 ts_tablet_manager.cc:1434] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.028s user 0.005s sys 0.002s
I20251026 14:58:31.466730 22848 tablet_bootstrap.cc:492] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:31.469055 22647 tablet_service.cc:1505] Processing CreateTablet for tablet 988275d9b2574d0381d8f21ee8d99381 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20251026 14:58:31.469574 22849 ts_tablet_manager.cc:1403] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.134s user 0.003s sys 0.010s
I20251026 14:58:31.470418 22647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 988275d9b2574d0381d8f21ee8d99381. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.472468 22849 raft_consensus.cc:359] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.474491 22659 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:31.475672 22774 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:31.476514 22855 raft_consensus.cc:2804] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:31.477407 22797 tablet_service.cc:1505] Processing CreateTablet for tablet 9440e0456e0648a5af21108f5fd7cfba (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20251026 14:58:31.480037 22850 tablet_bootstrap.cc:492] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:31.480702 22850 ts_tablet_manager.cc:1403] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.090s user 0.014s sys 0.004s
I20251026 14:58:31.481787 22855 raft_consensus.cc:697] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:31.482776 22855 consensus_queue.cc:237] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.486384 22798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea1e7da57b65433b9e01cad8c4adeb75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.490890 22645 tablet_service.cc:1505] Processing CreateTablet for tablet dea6ba9f8ecc4124bfedb6febafde72b (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20251026 14:58:31.491282 22791 tablet_service.cc:1505] Processing CreateTablet for tablet 988275d9b2574d0381d8f21ee8d99381 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20251026 14:58:31.492574 22791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 988275d9b2574d0381d8f21ee8d99381. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.495802 22797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9440e0456e0648a5af21108f5fd7cfba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.497997 22645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dea6ba9f8ecc4124bfedb6febafde72b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 14:58:31.489991 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 88343.2us
I20251026 14:58:31.497054 22646 tablet_service.cc:1505] Processing CreateTablet for tablet 9339edd1ecbf42e3a5e953b2b47f52f2 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20251026 14:58:31.500429 22646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9339edd1ecbf42e3a5e953b2b47f52f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.507884 22716 tablet_service.cc:1505] Processing CreateTablet for tablet e2b59eefda3b4f2d93934f02c4981c62 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20251026 14:58:31.509156 22716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2b59eefda3b4f2d93934f02c4981c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.509440 22848 tablet_bootstrap.cc:654] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:31.473193 22849 raft_consensus.cc:385] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:31.502960 22796 tablet_service.cc:1505] Processing CreateTablet for tablet 9339edd1ecbf42e3a5e953b2b47f52f2 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20251026 14:58:31.510771 22796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9339edd1ecbf42e3a5e953b2b47f52f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.504963 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:31.511581 22658 raft_consensus.cc:2393] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 1: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:31.485308 22850 raft_consensus.cc:359] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.506217 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:31.506580 22642 tablet_service.cc:1505] Processing CreateTablet for tablet 129989aafdc84d6b8e56fe97c7182fff (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20251026 14:58:31.516788 22734 raft_consensus.cc:2393] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 1: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:31.517485 22642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 129989aafdc84d6b8e56fe97c7182fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.503533 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:31.521021 22659 raft_consensus.cc:2393] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 1: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:58:31.525127 22556 catalog_manager.cc:5649] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:31.509953 22849 raft_consensus.cc:740] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:31.529031 22849 consensus_queue.cc:260] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.530699 22719 tablet_service.cc:1505] Processing CreateTablet for tablet 5a0e9c0787aa4053afa2791a86bb263a (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20251026 14:58:31.532052 22719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a0e9c0787aa4053afa2791a86bb263a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.532150 22724 tablet_service.cc:1505] Processing CreateTablet for tablet 82201a13e5354c8ea3b41436df180125 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20251026 14:58:31.533427 22724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82201a13e5354c8ea3b41436df180125. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.514281 22850 raft_consensus.cc:385] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:31.534879 22850 raft_consensus.cc:740] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:31.535617 22850 consensus_queue.cc:260] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.532814 22849 ts_tablet_manager.cc:1434] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.063s user 0.006s sys 0.000s
I20251026 14:58:31.542747 22850 ts_tablet_manager.cc:1434] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.062s user 0.009s sys 0.001s
I20251026 14:58:31.543622 22849 tablet_bootstrap.cc:492] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:31.551983 22848 tablet_bootstrap.cc:492] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:31.552532 22848 ts_tablet_manager.cc:1403] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.086s user 0.010s sys 0.000s
I20251026 14:58:31.553480 22720 tablet_service.cc:1505] Processing CreateTablet for tablet ea1e7da57b65433b9e01cad8c4adeb75 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20251026 14:58:31.554553 22849 tablet_bootstrap.cc:654] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:31.554828 22720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea1e7da57b65433b9e01cad8c4adeb75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.555549 22848 raft_consensus.cc:359] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.556272 22848 raft_consensus.cc:385] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:31.556555 22848 raft_consensus.cc:740] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:31.544698 22647 tablet_service.cc:1505] Processing CreateTablet for tablet 2979554f0ac542ab8cfcb32c7d2778d5 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20251026 14:58:31.543639 22850 tablet_bootstrap.cc:492] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:31.559513 22721 tablet_service.cc:1505] Processing CreateTablet for tablet dea6ba9f8ecc4124bfedb6febafde72b (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20251026 14:58:31.560787 22721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dea6ba9f8ecc4124bfedb6febafde72b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.561769 22854 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:31.561865 22795 tablet_service.cc:1505] Processing CreateTablet for tablet 2979554f0ac542ab8cfcb32c7d2778d5 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20251026 14:58:31.562294 22854 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.563210 22795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2979554f0ac542ab8cfcb32c7d2778d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.564323 22854 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:31.565881 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:31.568032 22648 tablet_service.cc:1505] Processing CreateTablet for tablet 32b237245fe346a689e59ddc315b4577 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
W20251026 14:58:31.569519 22622 leader_election.cc:343] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:31.570685 22723 tablet_service.cc:1505] Processing CreateTablet for tablet 9440e0456e0648a5af21108f5fd7cfba (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20251026 14:58:31.571120 22797 tablet_service.cc:1505] Processing CreateTablet for tablet c9edd43dec704b6e8351cebe3921d663 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20251026 14:58:31.572018 22723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9440e0456e0648a5af21108f5fd7cfba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.571750 22645 tablet_service.cc:1505] Processing CreateTablet for tablet c9edd43dec704b6e8351cebe3921d663 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20251026 14:58:31.572309 22797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9edd43dec704b6e8351cebe3921d663. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.575160 22717 tablet_service.cc:1505] Processing CreateTablet for tablet 988275d9b2574d0381d8f21ee8d99381 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20251026 14:58:31.576447 22717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 988275d9b2574d0381d8f21ee8d99381. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.578249 22715 tablet_service.cc:1505] Processing CreateTablet for tablet 129989aafdc84d6b8e56fe97c7182fff (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20251026 14:58:31.579568 22715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 129989aafdc84d6b8e56fe97c7182fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.578253 22718 tablet_service.cc:1505] Processing CreateTablet for tablet 9339edd1ecbf42e3a5e953b2b47f52f2 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20251026 14:58:31.581105 22718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9339edd1ecbf42e3a5e953b2b47f52f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.582275 22792 tablet_service.cc:1505] Processing CreateTablet for tablet e923fc1e1fa340a7b482357487981b39 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20251026 14:58:31.582335 22644 tablet_service.cc:1505] Processing CreateTablet for tablet 4e49f6058b03447ca17f7a94a8b071d8 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20251026 14:58:31.583612 22792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e923fc1e1fa340a7b482357487981b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.579222 22649 tablet_service.cc:1505] Processing CreateTablet for tablet e923fc1e1fa340a7b482357487981b39 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20251026 14:58:31.588918 22850 tablet_bootstrap.cc:654] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:31.590507 22793 tablet_service.cc:1505] Processing CreateTablet for tablet 4e49f6058b03447ca17f7a94a8b071d8 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20251026 14:58:31.591907 22793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e49f6058b03447ca17f7a94a8b071d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.594210 22641 tablet_service.cc:1505] Processing CreateTablet for tablet a673ec034b704e8e86556afe9322d8c2 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20251026 14:58:31.599205 22722 tablet_service.cc:1505] Processing CreateTablet for tablet c9edd43dec704b6e8351cebe3921d663 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20251026 14:58:31.600644 22722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9edd43dec704b6e8351cebe3921d663. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.603375 22641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a673ec034b704e8e86556afe9322d8c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.603183 22796 tablet_service.cc:1505] Processing CreateTablet for tablet a673ec034b704e8e86556afe9322d8c2 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20251026 14:58:31.604530 22796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a673ec034b704e8e86556afe9322d8c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.605957 22648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32b237245fe346a689e59ddc315b4577. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.606230 22791 tablet_service.cc:1505] Processing CreateTablet for tablet 32b237245fe346a689e59ddc315b4577 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20251026 14:58:31.607504 22791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32b237245fe346a689e59ddc315b4577. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.610535 22649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e923fc1e1fa340a7b482357487981b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.613754 22640 tablet_service.cc:1505] Processing CreateTablet for tablet 37e63cca2eb94ba8ac9b6c8cdf99c4a5 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20251026 14:58:31.615154 22640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37e63cca2eb94ba8ac9b6c8cdf99c4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.620122 22794 tablet_service.cc:1505] Processing CreateTablet for tablet 37e63cca2eb94ba8ac9b6c8cdf99c4a5 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20251026 14:58:31.621547 22794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37e63cca2eb94ba8ac9b6c8cdf99c4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.621650 22644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e49f6058b03447ca17f7a94a8b071d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.627571 22624 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:31.628808 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:31.629520 22734 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:31.629348 22719 tablet_service.cc:1505] Processing CreateTablet for tablet 32b237245fe346a689e59ddc315b4577 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20251026 14:58:31.630067 22854 raft_consensus.cc:2749] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251026 14:58:31.630774 22719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32b237245fe346a689e59ddc315b4577. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.633934 22645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9edd43dec704b6e8351cebe3921d663. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.634392 22643 tablet_service.cc:1505] Processing CreateTablet for tablet 21e9405b016a4df592291cba0e961b07 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20251026 14:58:31.635681 22643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21e9405b016a4df592291cba0e961b07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.637745 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:31.646126 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:31.646126 22716 tablet_service.cc:1505] Processing CreateTablet for tablet 2979554f0ac542ab8cfcb32c7d2778d5 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20251026 14:58:31.557415 22848 consensus_queue.cc:260] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.649909 22798 tablet_service.cc:1505] Processing CreateTablet for tablet 129989aafdc84d6b8e56fe97c7182fff (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20251026 14:58:31.651257 22798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 129989aafdc84d6b8e56fe97c7182fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.648566 22716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2979554f0ac542ab8cfcb32c7d2778d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.662016 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:31.662787 22700 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:31.667730 22647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2979554f0ac542ab8cfcb32c7d2778d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.676255 22848 ts_tablet_manager.cc:1434] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.123s user 0.008s sys 0.000s
I20251026 14:58:31.670033 22733 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:31.686183 22848 tablet_bootstrap.cc:492] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:31.690259 22700 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:31.691418 22856 raft_consensus.cc:2749] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251026 14:58:31.690599 22850 tablet_bootstrap.cc:492] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:31.692370 22850 ts_tablet_manager.cc:1403] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.149s user 0.007s sys 0.009s
I20251026 14:58:31.706110 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:58:31.708045 22624 leader_election.cc:343] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
W20251026 14:58:31.708323 22773 leader_election.cc:343] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:31.708549 22624 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:31.708832 22773 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:31.709926 22854 raft_consensus.cc:2749] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 14:58:31.710528 22773 leader_election.cc:343] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:31.710983 22773 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:31.711862 22878 raft_consensus.cc:2749] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:31.712471 22878 raft_consensus.cc:2749] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:31.712633 22848 tablet_bootstrap.cc:654] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:31.712658 22793 tablet_service.cc:1505] Processing CreateTablet for tablet 21e9405b016a4df592291cba0e961b07 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20251026 14:58:31.717341 22793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21e9405b016a4df592291cba0e961b07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.720428 22715 tablet_service.cc:1505] Processing CreateTablet for tablet 4e49f6058b03447ca17f7a94a8b071d8 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20251026 14:58:31.721787 22715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e49f6058b03447ca17f7a94a8b071d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.732957 22849 tablet_bootstrap.cc:492] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:31.739188 22722 tablet_service.cc:1505] Processing CreateTablet for tablet 37e63cca2eb94ba8ac9b6c8cdf99c4a5 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20251026 14:58:31.740612 22722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37e63cca2eb94ba8ac9b6c8cdf99c4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.741552 22849 ts_tablet_manager.cc:1403] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.198s user 0.022s sys 0.015s
I20251026 14:58:31.744577 22849 raft_consensus.cc:359] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.745525 22715 tablet_service.cc:1505] Processing CreateTablet for tablet 21e9405b016a4df592291cba0e961b07 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20251026 14:58:31.748330 22849 raft_consensus.cc:385] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:31.748699 22849 raft_consensus.cc:740] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:31.749853 22715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21e9405b016a4df592291cba0e961b07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.749492 22849 consensus_queue.cc:260] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.752404 22849 ts_tablet_manager.cc:1434] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20251026 14:58:31.753790 22849 tablet_bootstrap.cc:492] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:31.759138 22717 tablet_service.cc:1505] Processing CreateTablet for tablet a673ec034b704e8e86556afe9322d8c2 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20251026 14:58:31.766923 22855 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:31.767496 22855 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.761121 22717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a673ec034b704e8e86556afe9322d8c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.769570 22855 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:31.770524 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:31.771127 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:58:31.772063 22774 leader_election.cc:343] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
W20251026 14:58:31.772544 22773 leader_election.cc:343] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:31.772979 22773 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:31.774096 22855 raft_consensus.cc:2749] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:31.716955 22720 tablet_service.cc:1505] Processing CreateTablet for tablet e923fc1e1fa340a7b482357487981b39 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20251026 14:58:31.794193 22793 tablet_service.cc:1505] Processing CreateTablet for tablet 955c860c7f6f4f81bbe8e59e4c3dbcf5 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20251026 14:58:31.795580 22793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 955c860c7f6f4f81bbe8e59e4c3dbcf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.796487 22644 tablet_service.cc:1505] Processing CreateTablet for tablet 955c860c7f6f4f81bbe8e59e4c3dbcf5 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20251026 14:58:31.797859 22644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 955c860c7f6f4f81bbe8e59e4c3dbcf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.802259 22721 tablet_service.cc:1505] Processing CreateTablet for tablet 955c860c7f6f4f81bbe8e59e4c3dbcf5 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20251026 14:58:31.801205 22733 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:31.803597 22876 raft_consensus.cc:2749] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 14:58:31.804788 22604 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251026 14:58:31.814986 22848 tablet_bootstrap.cc:492] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:31.818166 22721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 955c860c7f6f4f81bbe8e59e4c3dbcf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.820820 22720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e923fc1e1fa340a7b482357487981b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.824528 22849 tablet_bootstrap.cc:654] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:31.826352 22850 raft_consensus.cc:359] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.827108 22850 raft_consensus.cc:385] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:31.827414 22850 raft_consensus.cc:740] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:31.819411 22848 ts_tablet_manager.cc:1403] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.138s user 0.031s sys 0.011s
I20251026 14:58:31.844739 22850 consensus_queue.cc:260] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.846848 22848 raft_consensus.cc:359] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.847627 22848 raft_consensus.cc:385] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:31.847929 22848 raft_consensus.cc:740] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:31.848745 22848 consensus_queue.cc:260] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.856197 22643 tablet_service.cc:1505] Processing CreateTablet for tablet 2a4022b371fc46c89fbc9a5470cda03e (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20251026 14:58:31.857190 22848 ts_tablet_manager.cc:1434] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.013s user 0.003s sys 0.003s
I20251026 14:58:31.857595 22643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a4022b371fc46c89fbc9a5470cda03e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.858261 22848 tablet_bootstrap.cc:492] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:31.866245 22850 ts_tablet_manager.cc:1434] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.173s user 0.004s sys 0.004s
I20251026 14:58:31.867388 22850 tablet_bootstrap.cc:492] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:31.875630 22854 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:31.876153 22854 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.877346 22715 tablet_service.cc:1505] Processing CreateTablet for tablet 2a4022b371fc46c89fbc9a5470cda03e (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20251026 14:58:31.878676 22715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a4022b371fc46c89fbc9a5470cda03e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.879628 22854 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:31.881076 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:31.881757 22808 raft_consensus.cc:2468] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 0.
I20251026 14:58:31.883117 22622 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:31.883044 22793 tablet_service.cc:1505] Processing CreateTablet for tablet 2a4022b371fc46c89fbc9a5470cda03e (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20251026 14:58:31.884493 22793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a4022b371fc46c89fbc9a5470cda03e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.885550 22848 tablet_bootstrap.cc:654] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:31.885685 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:58:31.887655 22624 leader_election.cc:343] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:31.894533 22854 raft_consensus.cc:2804] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:31.895057 22854 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:31.895478 22854 raft_consensus.cc:3060] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:31.899961 22794 tablet_service.cc:1505] Processing CreateTablet for tablet 627ebd70757a47b18827c38f071b5261 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20251026 14:58:31.901422 22850 tablet_bootstrap.cc:654] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:31.901108 22640 tablet_service.cc:1505] Processing CreateTablet for tablet 8bb7c83ff5ae4d4a802850a40803bc55 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20251026 14:58:31.901561 22794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 627ebd70757a47b18827c38f071b5261. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.902382 22640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bb7c83ff5ae4d4a802850a40803bc55. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.906503 22716 tablet_service.cc:1505] Processing CreateTablet for tablet 8bb7c83ff5ae4d4a802850a40803bc55 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20251026 14:58:31.907974 22716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bb7c83ff5ae4d4a802850a40803bc55. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.910441 22647 tablet_service.cc:1505] Processing CreateTablet for tablet 627ebd70757a47b18827c38f071b5261 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20251026 14:58:31.911759 22647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 627ebd70757a47b18827c38f071b5261. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.916980 22795 tablet_service.cc:1505] Processing CreateTablet for tablet 8bb7c83ff5ae4d4a802850a40803bc55 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20251026 14:58:31.917515 22644 tablet_service.cc:1505] Processing CreateTablet for tablet 5373ef47363d446fbf9d697ed507e754 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20251026 14:58:31.918565 22795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bb7c83ff5ae4d4a802850a40803bc55. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.918807 22644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5373ef47363d446fbf9d697ed507e754. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.921927 22854 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.923324 22724 tablet_service.cc:1505] Processing CreateTablet for tablet 5373ef47363d446fbf9d697ed507e754 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20251026 14:58:31.924690 22724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5373ef47363d446fbf9d697ed507e754. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.924732 22876 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:31.925587 22855 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:31.925452 22876 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.926087 22855 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.928095 22876 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:31.928097 22855 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:31.929222 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:31.929922 22659 raft_consensus.cc:2468] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:31.930107 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:58:31.931533 22697 leader_election.cc:343] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:31.931990 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:31.933468 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:58:31.934870 22773 leader_election.cc:343] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
W20251026 14:58:31.935814 22624 leader_election.cc:343] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:31.936422 22798 tablet_service.cc:1505] Processing CreateTablet for tablet 5373ef47363d446fbf9d697ed507e754 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20251026 14:58:31.936506 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:31.937628 22798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5373ef47363d446fbf9d697ed507e754. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 14:58:31.938377 22700 leader_election.cc:343] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:31.939013 22700 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:31.940737 22774 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:31.941836 22855 raft_consensus.cc:2804] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:31.942209 22855 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:31.941130 22854 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:31.943346 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:31.944046 22808 raft_consensus.cc:3060] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:31.944371 22876 raft_consensus.cc:2749] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:31.942582 22855 raft_consensus.cc:3060] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:31.966190 22722 tablet_service.cc:1505] Processing CreateTablet for tablet aceef9b41bab4ad3bf67b4d597a7fe11 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20251026 14:58:31.967666 22722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aceef9b41bab4ad3bf67b4d597a7fe11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.980020 22849 tablet_bootstrap.cc:492] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:31.980700 22849 ts_tablet_manager.cc:1403] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.227s user 0.015s sys 0.007s
I20251026 14:58:31.980744 22794 tablet_service.cc:1505] Processing CreateTablet for tablet aceef9b41bab4ad3bf67b4d597a7fe11 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20251026 14:58:31.982237 22794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aceef9b41bab4ad3bf67b4d597a7fe11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.983520 22643 tablet_service.cc:1505] Processing CreateTablet for tablet aceef9b41bab4ad3bf67b4d597a7fe11 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20251026 14:58:31.983719 22849 raft_consensus.cc:359] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.984508 22849 raft_consensus.cc:385] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:31.984830 22643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aceef9b41bab4ad3bf67b4d597a7fe11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:31.984788 22849 raft_consensus.cc:740] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:31.985649 22849 consensus_queue.cc:260] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:31.988065 22849 ts_tablet_manager.cc:1434] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251026 14:58:31.991557 22641 tablet_service.cc:1505] Processing CreateTablet for tablet 143c213a935041868ad3b0d9584ad863 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20251026 14:58:31.899961 22720 tablet_service.cc:1505] Processing CreateTablet for tablet 627ebd70757a47b18827c38f071b5261 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20251026 14:58:31.997071 22720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 627ebd70757a47b18827c38f071b5261. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:32.005908 22641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 143c213a935041868ad3b0d9584ad863. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:32.007280 22849 tablet_bootstrap.cc:492] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:32.008469 22791 tablet_service.cc:1505] Processing CreateTablet for tablet 143c213a935041868ad3b0d9584ad863 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20251026 14:58:32.009750 22791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 143c213a935041868ad3b0d9584ad863. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:32.031028 22855 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.041831 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:32.042704 22659 raft_consensus.cc:3060] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.052371 22848 tablet_bootstrap.cc:492] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:32.052891 22848 ts_tablet_manager.cc:1403] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.195s user 0.008s sys 0.016s
I20251026 14:58:31.998179 22721 tablet_service.cc:1505] Processing CreateTablet for tablet 143c213a935041868ad3b0d9584ad863 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20251026 14:58:32.049079 22647 tablet_service.cc:1505] Processing CreateTablet for tablet 5618ccace02e47938463dce9ca26e300 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20251026 14:58:32.055121 22647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5618ccace02e47938463dce9ca26e300. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:32.054927 22716 tablet_service.cc:1505] Processing CreateTablet for tablet 5618ccace02e47938463dce9ca26e300 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20251026 14:58:32.055881 22716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5618ccace02e47938463dce9ca26e300. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:32.059743 22878 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:32.063894 22850 tablet_bootstrap.cc:492] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:32.064587 22850 ts_tablet_manager.cc:1403] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.197s user 0.027s sys 0.015s
I20251026 14:58:32.064208 22878 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.066336 22878 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:32.068185 22849 tablet_bootstrap.cc:654] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:32.069382 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:32.069809 22721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 143c213a935041868ad3b0d9584ad863. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:32.070330 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:32.071296 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:32.071887 22733 raft_consensus.cc:2468] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
W20251026 14:58:32.073321 22773 leader_election.cc:343] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:32.074010 22773 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:58:32.075456 22774 leader_election.cc:343] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:32.076098 22878 consensus_queue.cc:1048] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:32.077320 22878 raft_consensus.cc:2804] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:32.077663 22878 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:32.077963 22878 raft_consensus.cc:3060] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.078339 22848 raft_consensus.cc:359] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.078886 22848 raft_consensus.cc:385] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:32.079128 22848 raft_consensus.cc:740] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:32.079694 22848 consensus_queue.cc:260] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.082242 22848 ts_tablet_manager.cc:1434] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.029s user 0.004s sys 0.004s
I20251026 14:58:32.082055 22791 tablet_service.cc:1505] Processing CreateTablet for tablet 5618ccace02e47938463dce9ca26e300 (DEFAULT_TABLE table=test-workload [id=0253264dd7b746d58adac3c9eeddb55d]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20251026 14:58:32.083164 22791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5618ccace02e47938463dce9ca26e300. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:32.083189 22848 tablet_bootstrap.cc:492] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:32.060187 22855 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:32.089915 22848 tablet_bootstrap.cc:654] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:32.119009 22848 tablet_bootstrap.cc:492] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:32.119530 22848 ts_tablet_manager.cc:1403] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.037s user 0.020s sys 0.007s
I20251026 14:58:32.126276 22849 tablet_bootstrap.cc:492] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:32.127005 22849 ts_tablet_manager.cc:1403] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.120s user 0.008s sys 0.013s
I20251026 14:58:32.129081 22808 raft_consensus.cc:2468] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:32.130909 22622 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:32.131143 22849 raft_consensus.cc:359] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.132081 22854 raft_consensus.cc:2804] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:32.132138 22849 raft_consensus.cc:385] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:32.132365 22659 raft_consensus.cc:2468] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:32.134377 22848 raft_consensus.cc:359] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.135143 22774 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:32.135639 22848 raft_consensus.cc:385] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:32.135982 22848 raft_consensus.cc:740] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:32.136690 22848 consensus_queue.cc:260] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.137477 22855 raft_consensus.cc:2804] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:32.132685 22849 raft_consensus.cc:740] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
W20251026 14:58:32.138598 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 64677.8us
I20251026 14:58:32.138724 22855 raft_consensus.cc:697] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:32.140003 22855 consensus_queue.cc:237] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.140262 22849 consensus_queue.cc:260] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.147149 22854 raft_consensus.cc:697] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:32.158229 22854 consensus_queue.cc:237] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.167987 22898 consensus_queue.cc:1048] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:32.168787 22849 ts_tablet_manager.cc:1434] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.041s user 0.009s sys 0.003s
I20251026 14:58:32.171176 22850 raft_consensus.cc:359] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.172446 22849 tablet_bootstrap.cc:492] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:32.172427 22850 raft_consensus.cc:385] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:32.172885 22850 raft_consensus.cc:740] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:32.182276 22878 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.186218 22849 tablet_bootstrap.cc:654] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:32.174561 22850 consensus_queue.cc:260] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.209219 22850 ts_tablet_manager.cc:1434] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.096s user 0.008s sys 0.000s
I20251026 14:58:32.209497 22878 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:32.210381 22850 tablet_bootstrap.cc:492] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:32.208577 22848 ts_tablet_manager.cc:1434] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.089s user 0.016s sys 0.034s
I20251026 14:58:32.214174 22848 tablet_bootstrap.cc:492] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:32.214500 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:58:32.217226 22774 leader_election.cc:343] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:32.218472 22876 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:32.219118 22876 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.219853 22850 tablet_bootstrap.cc:654] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
W20251026 14:58:32.223902 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 76732.7us
I20251026 14:58:32.227340 22553 catalog_manager.cc:5649] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:32.229652 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:32.230098 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:32.230913 22733 raft_consensus.cc:3060] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
W20251026 14:58:32.231642 22700 leader_election.cc:343] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:32.235450 22554 catalog_manager.cc:5649] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 0 to 1, leader changed from <none> to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 1 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:32.237596 22876 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:32.239625 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:32.240288 22808 raft_consensus.cc:2468] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:32.241842 22697 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:32.243511 22876 raft_consensus.cc:2804] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:32.244064 22876 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:32.244539 22876 raft_consensus.cc:3060] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.263561 22876 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.264550 22733 raft_consensus.cc:2468] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:32.268795 22773 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:32.270551 22849 tablet_bootstrap.cc:492] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:32.271239 22849 ts_tablet_manager.cc:1403] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.099s user 0.019s sys 0.000s
I20251026 14:58:32.271443 22891 raft_consensus.cc:2804] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:32.271970 22891 raft_consensus.cc:697] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:32.272778 22891 consensus_queue.cc:237] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.266487 22848 tablet_bootstrap.cc:654] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:32.274964 22849 raft_consensus.cc:359] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.275944 22849 raft_consensus.cc:385] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:32.276320 22849 raft_consensus.cc:740] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:32.277441 22849 consensus_queue.cc:260] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.283471 22849 ts_tablet_manager.cc:1434] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20251026 14:58:32.283995 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:32.284504 22849 tablet_bootstrap.cc:492] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:32.285105 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:32.286427 22808 raft_consensus.cc:3060] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.282541 22876 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:58:32.285246 22700 leader_election.cc:343] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:32.293099 22850 tablet_bootstrap.cc:492] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:32.294133 22850 ts_tablet_manager.cc:1403] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.084s user 0.027s sys 0.041s
I20251026 14:58:32.297063 22850 raft_consensus.cc:359] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:58:32.297837 22850 raft_consensus.cc:385] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:32.298151 22850 raft_consensus.cc:740] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:32.298054 22808 raft_consensus.cc:2468] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:32.299181 22850 consensus_queue.cc:260] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:58:32.299517 22697 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:32.300346 22876 raft_consensus.cc:2804] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:32.300789 22876 raft_consensus.cc:697] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:32.301631 22850 ts_tablet_manager.cc:1434] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20251026 14:58:32.302621 22850 tablet_bootstrap.cc:492] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:32.303098 22848 tablet_bootstrap.cc:492] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:32.303531 22848 ts_tablet_manager.cc:1403] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.090s user 0.009s sys 0.003s
I20251026 14:58:32.301663 22876 consensus_queue.cc:237] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.309507 22850 tablet_bootstrap.cc:654] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:32.305498 22554 catalog_manager.cc:5649] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:32.311916 22848 raft_consensus.cc:359] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.312664 22848 raft_consensus.cc:385] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:32.313006 22848 raft_consensus.cc:740] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:32.313941 22848 consensus_queue.cc:260] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.316378 22848 ts_tablet_manager.cc:1434] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.013s user 0.006s sys 0.002s
I20251026 14:58:32.317540 22848 tablet_bootstrap.cc:492] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:32.318586 22891 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:32.319068 22891 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.321235 22891 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:32.322712 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:32.323628 22734 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:32.324990 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:32.325663 22658 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:32.326443 22849 tablet_bootstrap.cc:654] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:32.326881 22773 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:32.328171 22891 raft_consensus.cc:2804] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:32.328527 22891 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:32.328852 22891 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.336301 22891 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.339335 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:32.339449 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:32.339982 22734 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.340248 22658 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.342499 22850 tablet_bootstrap.cc:492] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:32.342981 22850 ts_tablet_manager.cc:1403] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.009s
I20251026 14:58:32.345597 22850 raft_consensus.cc:359] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.346256 22850 raft_consensus.cc:385] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:32.346534 22850 raft_consensus.cc:740] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:32.347599 22850 consensus_queue.cc:260] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.349795 22850 ts_tablet_manager.cc:1434] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20251026 14:58:32.350687 22850 tablet_bootstrap.cc:492] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:32.351694 22658 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:32.353206 22774 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:32.354215 22899 raft_consensus.cc:2804] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:32.354702 22899 raft_consensus.cc:697] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:32.355492 22899 consensus_queue.cc:237] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.355926 22734 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:32.356173 22850 tablet_bootstrap.cc:654] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:32.361888 22848 tablet_bootstrap.cc:654] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:32.365608 22891 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:32.368008 22849 tablet_bootstrap.cc:492] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:32.368499 22849 ts_tablet_manager.cc:1403] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.084s user 0.010s sys 0.008s
I20251026 14:58:32.370954 22849 raft_consensus.cc:359] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.371635 22849 raft_consensus.cc:385] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:32.371898 22849 raft_consensus.cc:740] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:32.364316 22556 catalog_manager.cc:5649] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 0 to 1, leader changed from <none> to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:32.373425 22554 catalog_manager.cc:5649] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:32.374996 22848 tablet_bootstrap.cc:492] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:32.374871 22849 consensus_queue.cc:260] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.375461 22848 ts_tablet_manager.cc:1403] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.058s user 0.011s sys 0.007s
I20251026 14:58:32.375485 22850 tablet_bootstrap.cc:492] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:32.376238 22850 ts_tablet_manager.cc:1403] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.016s
I20251026 14:58:32.377652 22848 raft_consensus.cc:359] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.378180 22848 raft_consensus.cc:385] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:32.378428 22848 raft_consensus.cc:740] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:32.378690 22850 raft_consensus.cc:359] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.379338 22850 raft_consensus.cc:385] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:32.379611 22850 raft_consensus.cc:740] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:32.379124 22848 consensus_queue.cc:260] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.380244 22850 consensus_queue.cc:260] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.382261 22849 ts_tablet_manager.cc:1434] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.013s user 0.006s sys 0.002s
I20251026 14:58:32.382444 22850 ts_tablet_manager.cc:1434] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251026 14:58:32.383183 22849 tablet_bootstrap.cc:492] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:32.383185 22850 tablet_bootstrap.cc:492] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:32.385749 22848 ts_tablet_manager.cc:1434] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20251026 14:58:32.386670 22848 tablet_bootstrap.cc:492] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:32.389027 22850 tablet_bootstrap.cc:654] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:32.389035 22849 tablet_bootstrap.cc:654] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:32.392639 22848 tablet_bootstrap.cc:654] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
W20251026 14:58:32.427779 22685 tablet.cc:2378] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 14:58:32.443759 22849 tablet_bootstrap.cc:492] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:32.443989 22848 tablet_bootstrap.cc:492] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:32.444303 22850 tablet_bootstrap.cc:492] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:32.444303 22849 ts_tablet_manager.cc:1403] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.061s user 0.022s sys 0.037s
I20251026 14:58:32.444464 22848 ts_tablet_manager.cc:1403] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.058s user 0.021s sys 0.028s
I20251026 14:58:32.444919 22850 ts_tablet_manager.cc:1403] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.062s user 0.017s sys 0.032s
I20251026 14:58:32.446650 22848 raft_consensus.cc:359] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.447151 22848 raft_consensus.cc:385] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:32.447470 22848 raft_consensus.cc:740] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:32.447177 22849 raft_consensus.cc:359] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.447291 22850 raft_consensus.cc:359] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.448002 22849 raft_consensus.cc:385] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:32.448058 22850 raft_consensus.cc:385] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:32.448107 22848 consensus_queue.cc:260] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.448433 22849 raft_consensus.cc:740] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:32.448473 22850 raft_consensus.cc:740] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:32.449258 22850 consensus_queue.cc:260] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.449321 22849 consensus_queue.cc:260] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.450399 22848 ts_tablet_manager.cc:1434] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251026 14:58:32.451524 22848 tablet_bootstrap.cc:492] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:32.451529 22850 ts_tablet_manager.cc:1434] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20251026 14:58:32.452723 22850 tablet_bootstrap.cc:492] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:32.457792 22848 tablet_bootstrap.cc:654] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:32.458164 22850 tablet_bootstrap.cc:654] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:32.459128 22849 ts_tablet_manager.cc:1434] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20251026 14:58:32.459335 22854 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:32.460040 22849 tablet_bootstrap.cc:492] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:32.459810 22854 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.462177 22854 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:32.464251 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:32.465358 22734 raft_consensus.cc:2393] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 1: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:32.466352 22849 tablet_bootstrap.cc:654] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:32.467705 22899 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:32.467798 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:32.468235 22899 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.470644 22899 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:32.470682 22622 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:32.473096 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:32.473605 22897 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:32.474262 22897 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.475582 22658 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:32.479161 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:32.479307 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:32.479825 22733 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 0.
I20251026 14:58:32.481244 22624 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:32.481833 22774 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:32.482436 22895 raft_consensus.cc:3060] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.484397 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:32.485236 22808 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 0.
I20251026 14:58:32.486688 22899 raft_consensus.cc:2804] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:32.487172 22899 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:32.487591 22899 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.489526 22854 raft_consensus.cc:2804] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:32.490037 22854 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:32.490483 22854 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.477121 22897 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:32.499169 22895 raft_consensus.cc:2749] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:32.500682 22850 tablet_bootstrap.cc:492] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:32.501152 22850 ts_tablet_manager.cc:1403] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.049s user 0.012s sys 0.032s
I20251026 14:58:32.501549 22849 tablet_bootstrap.cc:492] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:32.501899 22849 ts_tablet_manager.cc:1403] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.042s user 0.011s sys 0.009s
I20251026 14:58:32.503753 22849 raft_consensus.cc:359] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.504245 22849 raft_consensus.cc:385] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:32.504691 22849 raft_consensus.cc:740] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:32.505250 22849 consensus_queue.cc:260] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.507424 22849 ts_tablet_manager.cc:1434] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20251026 14:58:32.508302 22876 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:32.508538 22849 tablet_bootstrap.cc:492] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:32.508780 22876 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:58:32.510776 22876 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:58:32.512547 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:32.512882 22854 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:32.514194 22700 leader_election.cc:343] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:32.514698 22849 tablet_bootstrap.cc:654] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:32.514847 22854 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:32.514779 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:32.504225 22899 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.516935 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:32.515558 22808 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:32.517640 22734 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.518656 22697 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:32.504177 22850 raft_consensus.cc:359] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.519230 22850 raft_consensus.cc:385] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:32.519510 22850 raft_consensus.cc:740] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:32.520185 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:32.520982 22876 raft_consensus.cc:2804] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:32.521394 22876 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:32.521826 22876 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.518774 22899 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:32.531145 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:32.532477 22808 raft_consensus.cc:2393] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 1: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:32.543251 22734 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:32.545006 22848 tablet_bootstrap.cc:492] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:32.545123 22624 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:32.545578 22848 ts_tablet_manager.cc:1403] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.094s user 0.011s sys 0.000s
I20251026 14:58:32.545468 22850 consensus_queue.cc:260] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.546377 22854 raft_consensus.cc:2804] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:32.548305 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:32.548892 22850 ts_tablet_manager.cc:1434] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.047s user 0.018s sys 0.012s
I20251026 14:58:32.548732 22848 raft_consensus.cc:359] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:58:32.549412 22848 raft_consensus.cc:385] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:32.549636 22848 raft_consensus.cc:740] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:32.550029 22850 tablet_bootstrap.cc:492] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:32.550230 22848 consensus_queue.cc:260] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:58:32.551550 22899 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:32.553229 22848 ts_tablet_manager.cc:1434] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20251026 14:58:32.552892 22899 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.554232 22849 tablet_bootstrap.cc:492] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:32.554175 22876 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:58:32.551869 22854 raft_consensus.cc:697] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:32.555737 22899 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:32.555783 22854 consensus_queue.cc:237] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.556687 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:32.554752 22849 ts_tablet_manager.cc:1403] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.046s user 0.017s sys 0.016s
I20251026 14:58:32.556842 22876 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:58:32.557883 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:32.558446 22659 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.557354 22734 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:32.560007 22849 raft_consensus.cc:359] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.560657 22849 raft_consensus.cc:385] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:32.560935 22849 raft_consensus.cc:740] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:32.563231 22850 tablet_bootstrap.cc:654] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:32.563331 22849 consensus_queue.cc:260] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.565654 22849 ts_tablet_manager.cc:1434] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20251026 14:58:32.566545 22849 tablet_bootstrap.cc:492] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:32.571738 22773 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:32.573879 22891 raft_consensus.cc:2804] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:32.574265 22891 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:32.574581 22891 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.576213 22849 tablet_bootstrap.cc:654] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:32.585616 22659 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:32.595510 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:32.596457 22658 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:32.598277 22553 catalog_manager.cc:5649] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 0 to 1, leader changed from <none> to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 1 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:32.600950 22848 tablet_bootstrap.cc:492] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:32.606436 22700 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:32.607551 22876 raft_consensus.cc:2804] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:32.607882 22899 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:32.608016 22876 raft_consensus.cc:697] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:32.608351 22899 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.608883 22876 consensus_queue.cc:237] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:58:32.611732 22774 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:32.612716 22899 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:32.613447 22899 raft_consensus.cc:2749] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251026 14:58:32.614207 22899 consensus_queue.cc:1048] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:32.614876 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:32.615473 22808 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.615672 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:32.616329 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:32.585161 22891 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:32.620605 22773 leader_election.cc:343] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:32.622535 22899 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:32.625509 22899 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.629772 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:32.630736 22734 raft_consensus.cc:2393] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 1: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:32.632340 22899 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:32.638876 22891 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:32.640180 22911 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:32.640889 22911 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.641480 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:32.642305 22734 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.646231 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:32.648752 22659 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:32.649170 22911 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:32.652108 22848 tablet_bootstrap.cc:654] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
W20251026 14:58:32.654338 22624 consensus_peers.cc:597] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 -> Peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Couldn't send request to peer 562bb1e1c8e445b8baecd29c3669658e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:32.659162 22700 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:32.659622 22734 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:32.660470 22911 raft_consensus.cc:2804] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:32.661006 22911 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:32.661492 22911 raft_consensus.cc:3060] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
W20251026 14:58:32.664662 22774 leader_election.cc:343] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:32.665480 22774 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:32.668023 22773 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:32.668931 22899 raft_consensus.cc:2804] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:32.667346 22891 raft_consensus.cc:2749] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:32.671051 22899 raft_consensus.cc:697] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:32.671952 22899 consensus_queue.cc:237] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.672294 22911 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.674217 22911 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:32.677222 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:32.678537 22659 raft_consensus.cc:3060] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.679579 22808 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:32.681119 22850 tablet_bootstrap.cc:492] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:32.681610 22850 ts_tablet_manager.cc:1403] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.132s user 0.051s sys 0.045s
I20251026 14:58:32.682199 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:32.683637 22657 raft_consensus.cc:2393] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 1: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:58:32.683458 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:32.685194 22656 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.686470 22734 raft_consensus.cc:3060] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.690626 22850 raft_consensus.cc:359] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.691346 22850 raft_consensus.cc:385] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:32.691665 22850 raft_consensus.cc:740] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:32.696856 22553 catalog_manager.cc:5649] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 0 to 1, leader changed from <none> to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:32.720841 22898 consensus_queue.cc:1048] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:32.722558 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:32.723220 22807 raft_consensus.cc:3060] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.726055 22774 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:32.731102 22849 tablet_bootstrap.cc:492] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:32.731612 22849 ts_tablet_manager.cc:1403] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.165s user 0.060s sys 0.050s
I20251026 14:58:32.730290 22850 consensus_queue.cc:260] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.724865 22876 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:32.732573 22898 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.739012 22659 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:32.739336 22656 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:32.739849 22850 ts_tablet_manager.cc:1434] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.058s user 0.018s sys 0.024s
I20251026 14:58:32.732331 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:32.744354 22849 raft_consensus.cc:359] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.739797 22876 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.748461 22850 tablet_bootstrap.cc:492] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:32.751041 22854 consensus_queue.cc:1048] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:32.751575 22700 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:32.749342 22878 consensus_queue.cc:1048] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:32.753444 22849 raft_consensus.cc:385] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:32.754012 22849 raft_consensus.cc:740] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:32.754995 22849 consensus_queue.cc:260] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.757153 22911 raft_consensus.cc:2804] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:32.771282 22849 ts_tablet_manager.cc:1434] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.039s user 0.004s sys 0.007s
I20251026 14:58:32.772646 22849 tablet_bootstrap.cc:492] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:32.782642 22876 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:32.806437 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:32.809609 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:32.818711 22848 tablet_bootstrap.cc:492] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:32.819391 22848 ts_tablet_manager.cc:1403] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.263s user 0.020s sys 0.006s
I20251026 14:58:32.811915 22911 raft_consensus.cc:697] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:32.829353 22854 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:32.829474 22697 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:32.829980 22854 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.832046 22854 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:32.833771 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:32.813634 22807 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:32.838500 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:32.838932 22848 raft_consensus.cc:359] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.839821 22848 raft_consensus.cc:385] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:32.840154 22848 raft_consensus.cc:740] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:32.842772 22806 raft_consensus.cc:2393] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 1: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:32.841527 22848 consensus_queue.cc:260] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:32.845497 22604 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251026 14:58:32.846694 22622 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:32.847738 22555 catalog_manager.cc:5649] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:32.854008 22848 ts_tablet_manager.cc:1434] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.034s user 0.008s sys 0.000s
I20251026 14:58:32.854812 22848 tablet_bootstrap.cc:492] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:32.851889 22895 raft_consensus.cc:3060] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.858608 22898 raft_consensus.cc:2749] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:32.862298 22936 consensus_queue.cc:1048] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20251026 14:58:32.852428 22911 consensus_queue.cc:237] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.887948 22850 tablet_bootstrap.cc:654] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:32.891891 22849 tablet_bootstrap.cc:654] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:32.895324 22935 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:32.896600 22935 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.899747 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:32.900419 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:32.900547 22806 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:32.902881 22697 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:32.904063 22937 raft_consensus.cc:2804] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:32.904472 22937 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:32.904868 22937 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
W20251026 14:58:32.908686 22700 leader_election.cc:343] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:32.907294 22935 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:32.910208 22895 raft_consensus.cc:2749] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:32.912703 22876 raft_consensus.cc:2749] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:32.913408 22934 consensus_queue.cc:1048] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20251026 14:58:32.926198 22876 consensus_queue.cc:1048] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:32.933961 22898 consensus_queue.cc:1048] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:32.938170 22876 consensus_queue.cc:1048] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:32.941062 22850 tablet_bootstrap.cc:492] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:32.941757 22850 ts_tablet_manager.cc:1403] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.194s user 0.007s sys 0.016s
I20251026 14:58:32.946080 22848 tablet_bootstrap.cc:654] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:32.948714 22850 raft_consensus.cc:359] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.949466 22850 raft_consensus.cc:385] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:32.949753 22850 raft_consensus.cc:740] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:32.951059 22850 consensus_queue.cc:260] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.951789 22937 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.959218 22850 ts_tablet_manager.cc:1434] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20251026 14:58:32.960625 22937 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:32.961723 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:58:32.963338 22700 leader_election.cc:343] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:32.961753 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:32.964596 22808 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.967041 22553 catalog_manager.cc:5649] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 0 to 1, leader changed from <none> to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:32.979076 22808 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:32.961442 22850 tablet_bootstrap.cc:492] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:32.981734 22895 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:32.984721 22895 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:32.986763 22895 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:32.988346 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:32.989071 22734 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 0.
I20251026 14:58:32.990675 22697 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:32.991096 22624 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:32.991618 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:32.991907 22895 raft_consensus.cc:2804] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:32.992228 22946 raft_consensus.cc:2804] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:32.992444 22895 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:32.992810 22946 raft_consensus.cc:697] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:32.993129 22895 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:32.993765 22946 consensus_queue.cc:237] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.001627 22808 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 0.
I20251026 14:58:33.005484 22937 consensus_queue.cc:1048] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:33.003011 22850 tablet_bootstrap.cc:654] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.007776 22848 tablet_bootstrap.cc:492] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:33.007758 22911 consensus_queue.cc:1048] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:33.009188 22895 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.011161 22895 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:33.012501 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:33.013324 22806 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:33.016120 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:33.016754 22733 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:33.028844 22848 ts_tablet_manager.cc:1403] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.174s user 0.015s sys 0.003s
I20251026 14:58:33.032644 22733 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:33.033958 22624 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:33.034833 22895 raft_consensus.cc:2804] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:33.035296 22895 raft_consensus.cc:697] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:33.036127 22895 consensus_queue.cc:237] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.046670 22806 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:33.047456 22849 tablet_bootstrap.cc:492] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:33.049552 22849 ts_tablet_manager.cc:1403] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.277s user 0.030s sys 0.005s
I20251026 14:58:33.050145 22848 raft_consensus.cc:359] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.050889 22848 raft_consensus.cc:385] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.051192 22848 raft_consensus.cc:740] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.051921 22848 consensus_queue.cc:260] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.055202 22849 raft_consensus.cc:359] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.055878 22849 raft_consensus.cc:385] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.056143 22849 raft_consensus.cc:740] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.056509 22850 tablet_bootstrap.cc:492] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:33.056936 22850 ts_tablet_manager.cc:1403] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.097s user 0.014s sys 0.025s
I20251026 14:58:33.060895 22848 ts_tablet_manager.cc:1434] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.031s user 0.005s sys 0.002s
I20251026 14:58:33.062377 22848 tablet_bootstrap.cc:492] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:33.056761 22849 consensus_queue.cc:260] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.067252 22849 ts_tablet_manager.cc:1434] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20251026 14:58:33.068241 22849 tablet_bootstrap.cc:492] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:33.070183 22555 catalog_manager.cc:5649] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 0 to 1, leader changed from <none> to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:33.074821 22850 raft_consensus.cc:359] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.075567 22850 raft_consensus.cc:385] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.075831 22850 raft_consensus.cc:740] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.076455 22850 consensus_queue.cc:260] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.078615 22850 ts_tablet_manager.cc:1434] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20251026 14:58:33.079510 22850 tablet_bootstrap.cc:492] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:33.079437 22554 catalog_manager.cc:5649] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 0 to 1, leader changed from <none> to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 1 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:33.088198 22850 tablet_bootstrap.cc:654] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.091573 22849 tablet_bootstrap.cc:654] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.092324 22848 tablet_bootstrap.cc:654] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.096732 22850 tablet_bootstrap.cc:492] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:33.097388 22850 ts_tablet_manager.cc:1403] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.004s
I20251026 14:58:33.099393 22848 tablet_bootstrap.cc:492] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:33.099936 22848 ts_tablet_manager.cc:1403] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.003s
I20251026 14:58:33.100075 22850 raft_consensus.cc:359] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.100800 22850 raft_consensus.cc:385] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.101105 22850 raft_consensus.cc:740] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.101750 22850 consensus_queue.cc:260] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.103102 22848 raft_consensus.cc:359] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.103730 22848 raft_consensus.cc:385] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.103837 22850 ts_tablet_manager.cc:1434] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251026 14:58:33.104031 22848 raft_consensus.cc:740] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.104799 22850 tablet_bootstrap.cc:492] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:33.104751 22848 consensus_queue.cc:260] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.107105 22848 ts_tablet_manager.cc:1434] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20251026 14:58:33.107273 22849 tablet_bootstrap.cc:492] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:33.107638 22849 ts_tablet_manager.cc:1403] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.040s user 0.014s sys 0.001s
I20251026 14:58:33.110209 22849 raft_consensus.cc:359] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.110915 22849 raft_consensus.cc:385] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.111210 22849 raft_consensus.cc:740] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.111761 22848 tablet_bootstrap.cc:492] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:33.111829 22849 consensus_queue.cc:260] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.114212 22849 ts_tablet_manager.cc:1434] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20251026 14:58:33.115204 22849 tablet_bootstrap.cc:492] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:33.117936 22848 tablet_bootstrap.cc:654] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.121991 22895 consensus_queue.cc:1048] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:33.126415 22849 tablet_bootstrap.cc:654] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.128298 22876 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:33.128944 22876 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.132654 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:33.132756 22895 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:33.133472 22895 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.134042 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:33.136037 22895 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:33.136762 22850 tablet_bootstrap.cc:654] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.136898 22876 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:33.136816 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:33.137142 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:33.137625 22733 raft_consensus.cc:2468] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:33.137737 22806 raft_consensus.cc:2468] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:33.138859 22624 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:33.140269 22697 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:33.141466 22937 raft_consensus.cc:2749] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:33.144963 22849 tablet_bootstrap.cc:492] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:33.145644 22849 ts_tablet_manager.cc:1403] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.031s user 0.015s sys 0.000s
I20251026 14:58:33.147357 22895 raft_consensus.cc:2804] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:33.147759 22895 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:33.148094 22895 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:33.149403 22850 tablet_bootstrap.cc:492] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:33.150019 22850 ts_tablet_manager.cc:1403] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.002s
I20251026 14:58:33.154261 22850 raft_consensus.cc:359] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.155300 22850 raft_consensus.cc:385] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.155658 22850 raft_consensus.cc:740] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.158479 22849 raft_consensus.cc:359] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.159528 22849 raft_consensus.cc:385] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.159888 22849 raft_consensus.cc:740] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.160935 22850 consensus_queue.cc:260] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.162569 22849 consensus_queue.cc:260] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.164217 22854 consensus_queue.cc:1048] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:33.170815 22850 ts_tablet_manager.cc:1434] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.020s user 0.009s sys 0.000s
I20251026 14:58:33.172215 22850 tablet_bootstrap.cc:492] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:33.180591 22850 tablet_bootstrap.cc:654] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.183928 22849 ts_tablet_manager.cc:1434] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.038s user 0.008s sys 0.001s
I20251026 14:58:33.185444 22849 tablet_bootstrap.cc:492] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:33.226997 22849 tablet_bootstrap.cc:654] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.228340 22876 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:33.229004 22876 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.232034 22878 consensus_queue.cc:1048] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:33.234310 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:33.235608 22876 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:58:33.235574 22700 leader_election.cc:343] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:33.240288 22878 consensus_queue.cc:1048] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:33.243794 22854 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:33.244377 22854 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.245260 22850 tablet_bootstrap.cc:492] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:33.245858 22850 ts_tablet_manager.cc:1403] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.074s user 0.026s sys 0.005s
I20251026 14:58:33.246448 22854 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:33.250916 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:33.251273 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:33.251863 22806 raft_consensus.cc:2468] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 0.
W20251026 14:58:33.253159 22624 leader_election.cc:343] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:33.188674 22895 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.253993 22622 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:33.256505 22849 tablet_bootstrap.cc:492] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:33.255247 22854 raft_consensus.cc:2804] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:33.256604 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:33.257205 22849 ts_tablet_manager.cc:1403] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.072s user 0.017s sys 0.012s
I20251026 14:58:33.257472 22732 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:33.259200 22895 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:33.258335 22854 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:33.260000 22850 raft_consensus.cc:359] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.261925 22850 raft_consensus.cc:385] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.262323 22850 raft_consensus.cc:740] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.261459 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:33.265094 22806 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:33.263296 22850 consensus_queue.cc:260] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.261354 22854 raft_consensus.cc:3060] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:33.270044 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:33.273103 22807 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:33.276249 22849 raft_consensus.cc:359] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.277238 22849 raft_consensus.cc:385] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.277781 22849 raft_consensus.cc:740] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.286088 22876 consensus_queue.cc:1048] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:33.280812 22849 consensus_queue.cc:260] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.290247 22697 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:33.300184 22850 ts_tablet_manager.cc:1434] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.054s user 0.006s sys 0.003s
I20251026 14:58:33.300376 22876 raft_consensus.cc:2804] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:33.301666 22850 tablet_bootstrap.cc:492] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:33.301432 22876 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:33.303092 22876 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:33.306507 22849 ts_tablet_manager.cc:1434] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.049s user 0.006s sys 0.002s
I20251026 14:58:33.306188 22854 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.308100 22854 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:33.309648 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:33.310340 22807 raft_consensus.cc:3060] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:33.317709 22733 raft_consensus.cc:3060] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:33.321494 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:58:33.324152 22624 leader_election.cc:343] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:33.335005 22807 raft_consensus.cc:2468] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:33.338364 22849 tablet_bootstrap.cc:492] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:33.339015 22622 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:33.336536 22732 raft_consensus.cc:2468] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:33.343657 22848 tablet_bootstrap.cc:492] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:33.344203 22848 ts_tablet_manager.cc:1403] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.233s user 0.056s sys 0.077s
I20251026 14:58:33.345855 22876 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.347249 22848 raft_consensus.cc:359] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.347927 22848 raft_consensus.cc:385] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.348193 22848 raft_consensus.cc:740] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.350056 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:58:33.351890 22700 leader_election.cc:343] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:33.352284 22854 raft_consensus.cc:2804] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:33.352820 22854 raft_consensus.cc:697] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:33.353971 22876 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:33.353780 22854 consensus_queue.cc:237] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.355479 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:33.356298 22807 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:33.366840 22850 tablet_bootstrap.cc:654] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.367835 22848 consensus_queue.cc:260] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.370463 22624 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:33.373534 22849 tablet_bootstrap.cc:654] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.374781 22895 raft_consensus.cc:2804] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:33.375275 22895 raft_consensus.cc:697] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:33.375758 22848 ts_tablet_manager.cc:1434] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.031s user 0.001s sys 0.005s
I20251026 14:58:33.376122 22895 consensus_queue.cc:237] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.367700 22806 raft_consensus.cc:2468] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:33.367400 22807 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:33.376693 22848 tablet_bootstrap.cc:492] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:33.387718 22697 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:33.390463 22876 raft_consensus.cc:2804] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:33.391355 22876 raft_consensus.cc:697] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:33.396813 22876 consensus_queue.cc:237] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:33.402058 22700 consensus_peers.cc:597] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e -> Peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Couldn't send request to peer df3a0d1619ee48e8ab68ee0543a72a29. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:33.410192 22848 tablet_bootstrap.cc:654] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.410704 22850 tablet_bootstrap.cc:492] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:33.411092 22850 ts_tablet_manager.cc:1403] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.110s user 0.007s sys 0.005s
I20251026 14:58:33.412871 22850 raft_consensus.cc:359] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.413522 22946 consensus_queue.cc:1048] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:33.414530 22849 tablet_bootstrap.cc:492] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:33.415469 22849 ts_tablet_manager.cc:1403] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.078s user 0.009s sys 0.002s
I20251026 14:58:33.417539 22553 catalog_manager.cc:5649] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 0 to 1, leader changed from <none> to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 1 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:33.418124 22849 raft_consensus.cc:359] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.418738 22849 raft_consensus.cc:385] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.419023 22849 raft_consensus.cc:740] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.419763 22849 consensus_queue.cc:260] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.419920 22553 catalog_manager.cc:5649] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 0 to 2, leader changed from <none> to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 2 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:33.413522 22850 raft_consensus.cc:385] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.422186 22850 raft_consensus.cc:740] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.423970 22850 consensus_queue.cc:260] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.425309 22850 ts_tablet_manager.cc:1434] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20251026 14:58:33.426015 22850 tablet_bootstrap.cc:492] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:33.428979 22849 ts_tablet_manager.cc:1434] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.013s user 0.007s sys 0.001s
I20251026 14:58:33.430022 22849 tablet_bootstrap.cc:492] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:33.430383 22848 tablet_bootstrap.cc:492] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:33.430886 22848 ts_tablet_manager.cc:1403] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.054s user 0.007s sys 0.006s
I20251026 14:58:33.431279 22850 tablet_bootstrap.cc:654] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.433617 22878 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:33.434116 22878 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.440629 22848 raft_consensus.cc:359] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.441956 22848 raft_consensus.cc:385] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.442293 22849 tablet_bootstrap.cc:654] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.444243 22895 consensus_queue.cc:1048] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:33.446134 22878 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:33.442291 22848 raft_consensus.cc:740] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.447393 22848 consensus_queue.cc:260] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.447695 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:58:33.449059 22773 leader_election.cc:343] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:33.449911 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:33.451360 22774 leader_election.cc:343] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:33.450841 22554 catalog_manager.cc:5649] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 0 to 1, leader changed from <none> to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:33.451809 22774 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:33.452564 22878 raft_consensus.cc:2749] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:33.449911 22848 ts_tablet_manager.cc:1434] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20251026 14:58:33.454682 22848 tablet_bootstrap.cc:492] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:33.455550 22849 tablet_bootstrap.cc:492] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:33.455981 22849 ts_tablet_manager.cc:1403] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.000s
I20251026 14:58:33.460129 22876 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:33.460526 22848 tablet_bootstrap.cc:654] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.461248 22850 tablet_bootstrap.cc:492] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:33.461771 22850 ts_tablet_manager.cc:1403] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.036s user 0.016s sys 0.016s
I20251026 14:58:33.458523 22849 raft_consensus.cc:359] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.462299 22849 raft_consensus.cc:385] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.460661 22876 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.463737 22849 raft_consensus.cc:740] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.464565 22849 consensus_queue.cc:260] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.465611 22876 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:33.466359 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:33.467098 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:33.467110 22849 ts_tablet_manager.cc:1434] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
W20251026 14:58:33.467896 22700 leader_election.cc:343] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
W20251026 14:58:33.469065 22697 leader_election.cc:343] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:33.469511 22697 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:33.468735 22849 tablet_bootstrap.cc:492] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:33.471072 22850 raft_consensus.cc:359] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.471736 22850 raft_consensus.cc:385] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.472015 22850 raft_consensus.cc:740] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.472643 22850 consensus_queue.cc:260] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.474713 22946 raft_consensus.cc:2749] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:33.474848 22850 ts_tablet_manager.cc:1434] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.013s user 0.004s sys 0.002s
I20251026 14:58:33.475730 22850 tablet_bootstrap.cc:492] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:33.477499 22849 tablet_bootstrap.cc:654] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.481518 22850 tablet_bootstrap.cc:654] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.483382 22946 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:33.484120 22848 tablet_bootstrap.cc:492] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:33.483925 22946 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.484709 22848 ts_tablet_manager.cc:1403] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.005s
I20251026 14:58:33.487331 22848 raft_consensus.cc:359] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.487989 22848 raft_consensus.cc:385] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.488255 22848 raft_consensus.cc:740] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.489444 22848 consensus_queue.cc:260] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.490365 22971 consensus_queue.cc:1048] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:33.490696 22849 tablet_bootstrap.cc:492] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:33.491240 22849 ts_tablet_manager.cc:1403] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.003s
I20251026 14:58:33.493896 22849 raft_consensus.cc:359] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.494654 22849 raft_consensus.cc:385] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.494956 22849 raft_consensus.cc:740] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.495805 22849 consensus_queue.cc:260] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.500367 22937 consensus_queue.cc:1048] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:33.502388 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:33.503294 22807 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:33.504434 22946 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:33.506839 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:33.507499 22659 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:33.508453 22848 ts_tablet_manager.cc:1434] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20251026 14:58:33.508816 22697 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:33.509405 22848 tablet_bootstrap.cc:492] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:33.510319 22946 raft_consensus.cc:2804] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:33.510730 22946 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:33.511121 22946 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:33.515909 22848 tablet_bootstrap.cc:654] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.517860 22849 ts_tablet_manager.cc:1434] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20251026 14:58:33.530022 22876 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:33.536885 22876 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.545689 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:33.546392 22659 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:33.549764 22850 tablet_bootstrap.cc:492] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:33.550271 22850 ts_tablet_manager.cc:1403] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.075s user 0.025s sys 0.036s
I20251026 14:58:33.551311 22700 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:33.552954 22850 raft_consensus.cc:359] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.553671 22850 raft_consensus.cc:385] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.553826 22911 raft_consensus.cc:2804] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:33.553958 22850 raft_consensus.cc:740] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.554229 22911 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:33.554579 22911 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:33.554605 22850 consensus_queue.cc:260] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.536484 22849 tablet_bootstrap.cc:492] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:33.561323 22876 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:33.562116 22850 ts_tablet_manager.cc:1434] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.012s user 0.008s sys 0.001s
I20251026 14:58:33.565032 22991 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:33.566617 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:33.568962 22850 tablet_bootstrap.cc:492] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:33.562880 22946 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.576336 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:33.576901 22659 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:33.567690 22991 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.577991 22878 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:33.578476 22878 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.582209 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:33.582489 22878 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:33.583910 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:33.586978 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:33.587862 22733 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:33.590270 22911 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.591197 22773 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:33.592281 22911 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:33.592746 22807 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:33.593490 22878 raft_consensus.cc:2804] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:33.593880 22878 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:33.594246 22878 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:33.594631 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:33.595242 22656 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
W20251026 14:58:33.595324 22774 leader_election.cc:343] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:33.597180 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:33.593889 22946 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:33.598842 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:33.600201 22991 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:33.601617 22808 raft_consensus.cc:2393] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 1: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:58:33.602433 22805 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:33.602459 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:33.608760 22804 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:33.613745 22656 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:33.615571 22700 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:33.617144 22805 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:33.617458 22991 raft_consensus.cc:2804] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:33.618238 22991 raft_consensus.cc:697] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:33.618827 22697 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:33.619588 22946 raft_consensus.cc:3060] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:33.620034 22804 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:33.622509 22848 tablet_bootstrap.cc:492] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:33.623417 22848 ts_tablet_manager.cc:1403] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.114s user 0.055s sys 0.011s
I20251026 14:58:33.623636 22659 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:33.619421 22991 consensus_queue.cc:237] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.627110 22700 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:33.628515 22990 raft_consensus.cc:2804] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:33.629000 22990 raft_consensus.cc:697] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:33.629199 22848 raft_consensus.cc:359] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.629948 22848 raft_consensus.cc:385] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.630292 22848 raft_consensus.cc:740] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.629870 22990 consensus_queue.cc:237] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.631971 22850 tablet_bootstrap.cc:654] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.633186 22848 consensus_queue.cc:260] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.639860 22937 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:33.640537 22937 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.646298 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:33.646572 22937 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:33.647205 22656 raft_consensus.cc:2468] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:33.650120 22700 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:33.651057 22937 raft_consensus.cc:2804] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:33.651427 22937 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:33.651736 22937 raft_consensus.cc:3060] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:33.645637 22946 raft_consensus.cc:2749] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:33.651988 22895 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:33.654165 22895 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.655995 22849 tablet_bootstrap.cc:654] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.657060 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:33.657680 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:33.658293 22805 raft_consensus.cc:2468] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:33.658349 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:33.659669 22622 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:33.660511 22971 raft_consensus.cc:2749] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:33.661163 22895 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:33.662211 22848 ts_tablet_manager.cc:1434] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.038s user 0.007s sys 0.000s
I20251026 14:58:33.662622 22937 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.665485 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:33.666105 22659 raft_consensus.cc:3060] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:33.667104 22937 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:33.662614 22878 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.668363 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:33.668970 22804 raft_consensus.cc:3060] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:33.670387 22878 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:33.671038 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:33.671396 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:33.674376 22733 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:33.675104 22659 raft_consensus.cc:2468] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:33.674522 22553 catalog_manager.cc:5649] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 0 to 1, leader changed from <none> to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:33.676272 22700 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:33.677130 22990 raft_consensus.cc:2804] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:33.677615 22990 raft_consensus.cc:697] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:33.681905 22804 raft_consensus.cc:2468] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:33.682242 22733 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
W20251026 14:58:33.683429 22774 leader_election.cc:343] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:33.682934 22848 tablet_bootstrap.cc:492] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:33.687075 22849 tablet_bootstrap.cc:492] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:33.687726 22849 ts_tablet_manager.cc:1403] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.151s user 0.007s sys 0.006s
I20251026 14:58:33.689137 22848 tablet_bootstrap.cc:654] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.678447 22990 consensus_queue.cc:237] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.690160 22773 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:33.691035 22849 raft_consensus.cc:359] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.691921 22849 raft_consensus.cc:385] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.692134 22849 raft_consensus.cc:740] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.692801 22849 consensus_queue.cc:260] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.693536 22878 raft_consensus.cc:2804] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:33.693734 22850 tablet_bootstrap.cc:492] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:33.694063 22878 raft_consensus.cc:697] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:33.694197 22850 ts_tablet_manager.cc:1403] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.130s user 0.019s sys 0.019s
I20251026 14:58:33.694802 22878 consensus_queue.cc:237] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.696771 22850 raft_consensus.cc:359] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.697475 22850 raft_consensus.cc:385] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.697772 22850 raft_consensus.cc:740] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.698676 22850 consensus_queue.cc:260] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.700853 22850 ts_tablet_manager.cc:1434] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251026 14:58:33.701745 22850 tablet_bootstrap.cc:492] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:33.705893 22849 ts_tablet_manager.cc:1434] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.018s user 0.009s sys 0.000s
I20251026 14:58:33.707024 22849 tablet_bootstrap.cc:492] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:33.714300 22553 catalog_manager.cc:5649] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 0 to 1, leader changed from <none> to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:33.715929 22553 catalog_manager.cc:5649] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 0 to 1, leader changed from <none> to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:33.716929 22555 catalog_manager.cc:5649] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:33.725521 22849 tablet_bootstrap.cc:654] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.726454 22850 tablet_bootstrap.cc:654] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.731096 22848 tablet_bootstrap.cc:492] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:33.732429 22848 ts_tablet_manager.cc:1403] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.050s user 0.006s sys 0.006s
I20251026 14:58:33.735296 22848 raft_consensus.cc:359] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.735765 22848 raft_consensus.cc:385] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.735973 22848 raft_consensus.cc:740] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.736465 22848 consensus_queue.cc:260] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.737993 22848 ts_tablet_manager.cc:1434] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251026 14:58:33.740010 22848 tablet_bootstrap.cc:492] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:33.740432 22850 tablet_bootstrap.cc:492] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:33.740756 22849 tablet_bootstrap.cc:492] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:33.740875 22850 ts_tablet_manager.cc:1403] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.039s user 0.008s sys 0.002s
I20251026 14:58:33.741329 22849 ts_tablet_manager.cc:1403] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.004s
I20251026 14:58:33.743428 22850 raft_consensus.cc:359] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.744365 22850 raft_consensus.cc:385] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.744649 22850 raft_consensus.cc:740] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.745231 22850 consensus_queue.cc:260] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.745755 22848 tablet_bootstrap.cc:654] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.743888 22849 raft_consensus.cc:359] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.747813 22849 raft_consensus.cc:385] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.748098 22849 raft_consensus.cc:740] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.748814 22849 consensus_queue.cc:260] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.751015 22849 ts_tablet_manager.cc:1434] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20251026 14:58:33.751403 22850 ts_tablet_manager.cc:1434] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.010s user 0.003s sys 0.003s
I20251026 14:58:33.752310 22850 tablet_bootstrap.cc:492] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:33.759168 22850 tablet_bootstrap.cc:654] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.759284 22849 tablet_bootstrap.cc:492] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:33.764088 22848 tablet_bootstrap.cc:492] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:33.764596 22848 ts_tablet_manager.cc:1403] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.010s
I20251026 14:58:33.767071 22849 tablet_bootstrap.cc:654] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.767141 22848 raft_consensus.cc:359] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.767793 22850 tablet_bootstrap.cc:492] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:33.767867 22848 raft_consensus.cc:385] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.768276 22848 raft_consensus.cc:740] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.768383 22850 ts_tablet_manager.cc:1403] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20251026 14:58:33.768930 22848 consensus_queue.cc:260] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.771176 22848 ts_tablet_manager.cc:1434] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20251026 14:58:33.770969 22850 raft_consensus.cc:359] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.771644 22850 raft_consensus.cc:385] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.771931 22850 raft_consensus.cc:740] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.775784 22850 consensus_queue.cc:260] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.778030 22848 tablet_bootstrap.cc:492] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:33.780532 22850 ts_tablet_manager.cc:1434] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20251026 14:58:33.781191 22849 tablet_bootstrap.cc:492] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:33.781579 22849 ts_tablet_manager.cc:1403] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.007s
I20251026 14:58:33.783313 22849 raft_consensus.cc:359] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.784073 22849 raft_consensus.cc:385] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.784278 22850 tablet_bootstrap.cc:492] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:33.784350 22849 raft_consensus.cc:740] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.785181 22849 consensus_queue.cc:260] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.787333 22849 ts_tablet_manager.cc:1434] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251026 14:58:33.784232 22848 tablet_bootstrap.cc:654] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.791298 22895 consensus_queue.cc:1048] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:33.794963 22850 tablet_bootstrap.cc:654] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.798516 22849 tablet_bootstrap.cc:492] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:33.804556 22848 tablet_bootstrap.cc:492] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:33.805119 22848 ts_tablet_manager.cc:1403] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.013s
I20251026 14:58:33.806166 22849 tablet_bootstrap.cc:654] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.807716 22848 raft_consensus.cc:359] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.808403 22848 raft_consensus.cc:385] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.808668 22848 raft_consensus.cc:740] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.809299 22848 consensus_queue.cc:260] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.815220 22897 consensus_queue.cc:1048] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:33.818497 22848 ts_tablet_manager.cc:1434] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20251026 14:58:33.819542 22848 tablet_bootstrap.cc:492] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:33.822182 22897 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:33.822698 22897 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.824699 22897 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:33.830394 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:33.831281 22805 raft_consensus.cc:2393] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 1: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:33.831764 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:33.833642 22624 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:33.835562 22897 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:33.840749 22848 tablet_bootstrap.cc:654] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.846220 22897 raft_consensus.cc:2749] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:33.848443 22849 tablet_bootstrap.cc:492] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:33.848840 22849 ts_tablet_manager.cc:1403] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.051s user 0.028s sys 0.016s
I20251026 14:58:33.851208 22849 raft_consensus.cc:359] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.851882 22849 raft_consensus.cc:385] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.852144 22849 raft_consensus.cc:740] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.852787 22849 consensus_queue.cc:260] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.854995 22849 ts_tablet_manager.cc:1434] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20251026 14:58:33.855886 22849 tablet_bootstrap.cc:492] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:33.856725 22848 tablet_bootstrap.cc:492] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:33.857257 22848 ts_tablet_manager.cc:1403] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.005s
I20251026 14:58:33.860100 22848 raft_consensus.cc:359] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.860859 22848 raft_consensus.cc:385] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.861199 22848 raft_consensus.cc:740] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.861660 22849 tablet_bootstrap.cc:654] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.862044 22848 consensus_queue.cc:260] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.864272 22897 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:33.864709 22897 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.867630 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:58:33.869063 22624 leader_election.cc:343] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
W20251026 14:58:33.869681 22604 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251026 14:58:33.870796 22878 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:33.870146 22897 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:33.871320 22878 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.872269 22848 ts_tablet_manager.cc:1434] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20251026 14:58:33.873356 22848 tablet_bootstrap.cc:492] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:33.873313 22878 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:33.874586 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:33.875211 22805 raft_consensus.cc:2468] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 0.
I20251026 14:58:33.876856 22849 tablet_bootstrap.cc:492] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:33.877183 22850 tablet_bootstrap.cc:492] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:33.877336 22849 ts_tablet_manager.cc:1403] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.005s
I20251026 14:58:33.877612 22850 ts_tablet_manager.cc:1403] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.094s user 0.008s sys 0.004s
I20251026 14:58:33.879184 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:33.880012 22849 raft_consensus.cc:359] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.880923 22849 raft_consensus.cc:385] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251026 14:58:33.880879 22773 leader_election.cc:343] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:33.883476 22898 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:33.884132 22898 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.887357 22850 raft_consensus.cc:359] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.887827 22898 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:33.888108 22850 raft_consensus.cc:385] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.881852 22849 raft_consensus.cc:740] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.888424 22850 raft_consensus.cc:740] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.888191 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:33.888823 22848 tablet_bootstrap.cc:654] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.889043 22849 consensus_queue.cc:260] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.889158 22850 consensus_queue.cc:260] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:33.889976 22773 leader_election.cc:343] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:33.891307 22850 ts_tablet_manager.cc:1434] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20251026 14:58:33.891422 22849 ts_tablet_manager.cc:1434] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20251026 14:58:33.892740 22850 tablet_bootstrap.cc:492] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:33.893246 22990 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:33.893877 22990 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.897408 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:33.899497 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:33.902032 22850 tablet_bootstrap.cc:654] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.901418 22990 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:58:33.903184 22697 leader_election.cc:343] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
W20251026 14:58:33.907399 22700 leader_election.cc:343] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:33.908041 22700 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:33.908387 22878 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:33.909618 22878 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.910416 22895 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:33.911064 22895 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.912057 22849 tablet_bootstrap.cc:492] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:33.912020 22878 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:33.908993 22850 tablet_bootstrap.cc:492] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:33.912829 22850 ts_tablet_manager.cc:1403] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.001s
I20251026 14:58:33.910660 22622 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:33.921043 22970 raft_consensus.cc:2804] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:33.921550 22970 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:33.921993 22970 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:33.922578 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:33.923805 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:33.925974 22878 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:33.927971 22805 raft_consensus.cc:1275] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
W20251026 14:58:33.929461 22774 leader_election.cc:343] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:33.929903 22774 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:33.930068 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:33.930987 22898 raft_consensus.cc:2749] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:33.925992 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:33.926642 22878 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.915849 22850 raft_consensus.cc:359] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.918944 22895 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:33.920400 22849 tablet_bootstrap.cc:654] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:33.920212 22990 raft_consensus.cc:2749] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:33.935628 22804 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 0.
I20251026 14:58:33.936269 22850 raft_consensus.cc:385] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.940971 22850 raft_consensus.cc:740] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.938558 22878 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:33.941810 22850 consensus_queue.cc:260] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.944751 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:58:33.949334 22774 leader_election.cc:343] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:33.950834 22848 tablet_bootstrap.cc:492] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:33.936167 22897 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:33.952201 22897 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.951676 22848 ts_tablet_manager.cc:1403] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.079s user 0.019s sys 0.040s
I20251026 14:58:33.954262 22850 ts_tablet_manager.cc:1434] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.041s user 0.003s sys 0.003s
I20251026 14:58:33.955204 22850 tablet_bootstrap.cc:492] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:33.951658 23013 consensus_queue.cc:1048] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20251026 14:58:33.958794 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:33.950888 22733 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 0.
I20251026 14:58:33.959973 22658 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:33.962658 22732 raft_consensus.cc:1275] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251026 14:58:33.964416 22849 tablet_bootstrap.cc:492] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:33.964872 22849 ts_tablet_manager.cc:1403] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.053s user 0.013s sys 0.003s
I20251026 14:58:33.959275 22622 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:33.969705 22774 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:33.970515 22878 raft_consensus.cc:2804] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:33.970394 22848 raft_consensus.cc:359] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.970959 22878 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:33.971128 22848 raft_consensus.cc:385] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:33.971371 22878 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:33.971460 22848 raft_consensus.cc:740] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:33.972167 22848 consensus_queue.cc:260] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:33.977504 23013 raft_consensus.cc:2804] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:33.979728 22897 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:33.981607 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:33.981932 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:33.981994 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:33.982594 22730 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:33.982901 22734 raft_consensus.cc:2393] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 1: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:33.984299 22878 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:33.985265 22773 leader_election.cc:343] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:33.985759 22773 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:33.987876 22898 raft_consensus.cc:2749] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:33.987984 23013 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:33.988397 23013 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:33.989528 22991 consensus_queue.cc:1048] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:33.989861 22878 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:33.989877 22658 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.001039 22848 ts_tablet_manager.cc:1434] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.047s user 0.006s sys 0.000s
I20251026 14:58:34.002286 22848 tablet_bootstrap.cc:492] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:34.002511 22854 consensus_queue.cc:1048] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:33.997885 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:34.004401 22734 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.006466 22937 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:34.006951 22937 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.009425 22854 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:34.008864 22937 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:34.009913 22854 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.014073 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:34.016894 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:34.017213 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:34.015720 22854 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:34.022485 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:34.022550 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:34.023278 22657 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:34.023459 22806 raft_consensus.cc:2468] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 0.
I20251026 14:58:34.024777 22622 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:34.028331 22895 raft_consensus.cc:2804] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:34.028764 22895 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:58:34.030140 22697 leader_election.cc:343] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:34.029121 22895 raft_consensus.cc:3060] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.031100 22848 tablet_bootstrap.cc:654] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:34.037146 22849 raft_consensus.cc:359] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.040527 22970 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.042552 22656 raft_consensus.cc:2393] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 1: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:58:34.043787 22734 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:34.045017 22774 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:34.045928 22878 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.048293 22895 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.053656 22849 raft_consensus.cc:385] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:34.055680 22895 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:34.056677 22773 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:34.057559 22898 raft_consensus.cc:2804] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:34.058524 22898 raft_consensus.cc:697] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:34.054637 22849 raft_consensus.cc:740] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:34.060691 22991 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:58:34.053818 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 51690.3us
I20251026 14:58:34.059839 22849 consensus_queue.cc:260] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.061133 22991 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.061158 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:34.070673 22806 raft_consensus.cc:3060] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.071271 22991 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:34.072436 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:34.062217 22898 consensus_queue.cc:237] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.076759 22806 raft_consensus.cc:2468] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:34.064410 22970 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:34.066535 22878 raft_consensus.cc:2749] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:34.067624 22937 consensus_queue.cc:1048] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:34.067968 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:34.085695 22896 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:34.080621 23013 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.082654 22622 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:34.083112 22849 ts_tablet_manager.cc:1434] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.118s user 0.009s sys 0.000s
I20251026 14:58:34.086994 22850 tablet_bootstrap.cc:654] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:34.082110 22700 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:34.086388 22896 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.092321 22896 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:34.088891 22624 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:34.094101 22971 raft_consensus.cc:2804] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:34.094561 22971 raft_consensus.cc:697] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:34.094672 22897 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.095417 22971 consensus_queue.cc:237] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.098253 22991 raft_consensus.cc:2804] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:34.098664 22991 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:34.099012 22991 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.089346 22849 tablet_bootstrap.cc:492] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:34.102239 22937 consensus_queue.cc:1048] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:34.103665 22937 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:34.104141 22937 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.107568 22659 raft_consensus.cc:2393] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 1: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:58:34.104190 22897 raft_consensus.cc:2749] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:34.109236 22937 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:34.112316 22804 raft_consensus.cc:2468] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:34.123999 22991 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.127631 22991 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:34.118736 22878 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:34.189680 22555 catalog_manager.cc:5649] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:34.193483 22553 catalog_manager.cc:5649] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 0 to 1, leader changed from <none> to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 1 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:34.129539 23013 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:34.179678 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:34.140622 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:34.131721 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:34.130753 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:34.199149 22730 raft_consensus.cc:2468] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 0.
I20251026 14:58:34.138585 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:34.199996 22733 raft_consensus.cc:3060] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.196345 22622 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:34.193032 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:34.189769 22878 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.210006 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:34.205466 23013 raft_consensus.cc:2804] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:34.211531 23013 raft_consensus.cc:697] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:34.212394 23013 consensus_queue.cc:237] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.213055 22731 raft_consensus.cc:2393] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 1: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:34.215602 22733 raft_consensus.cc:2468] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:34.218708 22624 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:34.220790 22897 raft_consensus.cc:2749] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251026 14:58:34.205099 22732 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.225827 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:34.227456 22850 tablet_bootstrap.cc:492] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:34.230554 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:34.231174 22700 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:34.231091 22991 consensus_queue.cc:1048] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:34.232152 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:34.232748 22659 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.232867 22991 raft_consensus.cc:2749] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:34.230358 22850 ts_tablet_manager.cc:1403] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.275s user 0.006s sys 0.015s
W20251026 14:58:34.237192 22773 leader_election.cc:343] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
W20251026 14:58:34.238138 22774 leader_election.cc:343] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:34.238587 22774 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:34.242767 22848 tablet_bootstrap.cc:492] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:34.243249 22848 ts_tablet_manager.cc:1403] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.241s user 0.013s sys 0.017s
I20251026 14:58:34.244568 22898 raft_consensus.cc:2749] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 14:58:34.248179 22700 leader_election.cc:343] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:34.258862 23020 consensus_queue.cc:1048] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
I20251026 14:58:34.257931 22848 raft_consensus.cc:359] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.262429 22848 raft_consensus.cc:385] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:34.263307 22848 raft_consensus.cc:740] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:34.264303 22848 consensus_queue.cc:260] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.274663 22553 catalog_manager.cc:5649] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 0 to 1, leader changed from <none> to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 1 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:34.276253 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:34.278270 22734 raft_consensus.cc:2393] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 1: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:34.280081 22848 ts_tablet_manager.cc:1434] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.037s user 0.007s sys 0.000s
I20251026 14:58:34.281116 22848 tablet_bootstrap.cc:492] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:34.274255 22878 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:34.269518 22732 raft_consensus.cc:2468] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:34.288235 22850 raft_consensus.cc:359] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.291798 22850 raft_consensus.cc:385] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:34.292105 22850 raft_consensus.cc:740] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:34.292613 22850 consensus_queue.cc:260] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.294473 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:34.297873 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:34.299304 22850 ts_tablet_manager.cc:1434] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.064s user 0.003s sys 0.003s
I20251026 14:58:34.300536 22850 tablet_bootstrap.cc:492] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:34.301462 22937 consensus_queue.cc:1048] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:34.298709 22805 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.305723 22849 tablet_bootstrap.cc:654] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:34.312377 22659 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:34.314286 22911 consensus_queue.cc:1048] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251026 14:58:34.320055 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:34.321789 22657 raft_consensus.cc:2393] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 1: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:34.323621 22805 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
W20251026 14:58:34.320827 22697 leader_election.cc:343] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:34.324471 22697 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:34.327875 22990 raft_consensus.cc:2749] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:34.328810 22700 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:34.329319 22991 consensus_queue.cc:1048] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:34.329870 22946 consensus_queue.cc:1048] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:34.330796 22991 raft_consensus.cc:2804] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:34.331276 22991 raft_consensus.cc:697] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:34.332252 22991 consensus_queue.cc:237] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.331163 22774 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:34.349412 22848 tablet_bootstrap.cc:654] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:34.349692 22898 raft_consensus.cc:2749] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:34.383661 22850 tablet_bootstrap.cc:654] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
W20251026 14:58:34.382863 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 53986.3us
I20251026 14:58:34.390173 22991 raft_consensus.cc:493] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:34.390722 22991 raft_consensus.cc:515] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.392589 22991 leader_election.cc:290] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:34.396250 23021 consensus_queue.cc:1048] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20251026 14:58:34.399689 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:34.413678 22555 catalog_manager.cc:5649] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 0 to 1, leader changed from <none> to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:34.426260 22849 tablet_bootstrap.cc:492] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:34.426694 22849 ts_tablet_manager.cc:1403] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.338s user 0.023s sys 0.000s
W20251026 14:58:34.415431 22697 leader_election.cc:343] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:34.433054 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:34.434610 22700 leader_election.cc:343] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:34.435267 22700 leader_election.cc:304] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:34.450246 23031 raft_consensus.cc:2749] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:34.455854 22849 raft_consensus.cc:359] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.456669 22849 raft_consensus.cc:385] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:34.456948 22849 raft_consensus.cc:740] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:34.457973 22849 consensus_queue.cc:260] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.463634 22849 ts_tablet_manager.cc:1434] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.037s user 0.006s sys 0.000s
I20251026 14:58:34.464613 22849 tablet_bootstrap.cc:492] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:34.466130 22848 tablet_bootstrap.cc:492] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:34.466591 22848 ts_tablet_manager.cc:1403] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.186s user 0.009s sys 0.008s
I20251026 14:58:34.470372 22848 raft_consensus.cc:359] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.471346 22848 raft_consensus.cc:385] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:34.473662 22848 raft_consensus.cc:740] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:34.472467 22850 tablet_bootstrap.cc:492] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:34.474532 22850 ts_tablet_manager.cc:1403] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.174s user 0.010s sys 0.018s
I20251026 14:58:34.486497 22849 tablet_bootstrap.cc:654] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:34.499177 22850 raft_consensus.cc:359] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.500481 22848 consensus_queue.cc:260] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.501303 22850 raft_consensus.cc:385] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:34.501672 22850 raft_consensus.cc:740] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:34.502480 22850 consensus_queue.cc:260] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.505025 22850 ts_tablet_manager.cc:1434] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.030s user 0.005s sys 0.000s
I20251026 14:58:34.506829 22850 tablet_bootstrap.cc:492] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:34.507619 22849 tablet_bootstrap.cc:492] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:34.508023 22849 ts_tablet_manager.cc:1403] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.044s user 0.015s sys 0.010s
I20251026 14:58:34.509920 22849 raft_consensus.cc:359] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.510578 22849 raft_consensus.cc:385] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:34.510847 22849 raft_consensus.cc:740] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:34.511464 22849 consensus_queue.cc:260] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.511835 22848 ts_tablet_manager.cc:1434] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.045s user 0.005s sys 0.003s
I20251026 14:58:34.512603 22850 tablet_bootstrap.cc:654] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:34.513003 22848 tablet_bootstrap.cc:492] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:34.513729 22849 ts_tablet_manager.cc:1434] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20251026 14:58:34.520555 22849 tablet_bootstrap.cc:492] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:34.527102 22849 tablet_bootstrap.cc:654] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:34.532305 22848 tablet_bootstrap.cc:654] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:34.538708 22850 tablet_bootstrap.cc:492] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:34.539219 22850 ts_tablet_manager.cc:1403] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.012s
I20251026 14:58:34.539389 22849 tablet_bootstrap.cc:492] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:34.539865 22849 ts_tablet_manager.cc:1403] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.004s
I20251026 14:58:34.542411 22849 raft_consensus.cc:359] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.543118 22849 raft_consensus.cc:385] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:34.542976 22850 raft_consensus.cc:359] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.543457 22849 raft_consensus.cc:740] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:34.543833 22850 raft_consensus.cc:385] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:34.544174 22850 raft_consensus.cc:740] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:34.544430 22849 consensus_queue.cc:260] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.544942 22850 consensus_queue.cc:260] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.546171 22848 tablet_bootstrap.cc:492] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:34.550077 22898 consensus_queue.cc:1048] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:34.551090 22850 ts_tablet_manager.cc:1434] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.012s user 0.006s sys 0.002s
I20251026 14:58:34.552044 22850 tablet_bootstrap.cc:492] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:34.554807 22849 ts_tablet_manager.cc:1434] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.015s user 0.003s sys 0.004s
I20251026 14:58:34.555826 22849 tablet_bootstrap.cc:492] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:34.559557 22850 tablet_bootstrap.cc:654] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:34.560225 23013 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:34.560758 23013 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.561847 22849 tablet_bootstrap.cc:654] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:34.564272 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:34.564966 22806 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 0.
I20251026 14:58:34.565128 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:34.565774 23016 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:34.566352 22622 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:34.566431 22731 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 0.
I20251026 14:58:34.566439 23016 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.567188 22971 raft_consensus.cc:2804] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:34.567623 22971 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:34.568078 22971 raft_consensus.cc:3060] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.568564 23013 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:34.573814 22898 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:34.574393 22898 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.575428 23016 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:34.576382 22898 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:34.576521 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:34.577172 22806 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 0.
I20251026 14:58:34.577975 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:34.578085 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:34.578274 22622 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:34.578722 22734 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:34.579537 22850 tablet_bootstrap.cc:492] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:34.579493 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:34.580130 22850 ts_tablet_manager.cc:1403] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.005s
I20251026 14:58:34.580215 22773 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:34.580263 22658 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:34.581204 22898 raft_consensus.cc:2804] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:34.581621 22898 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:34.582005 22898 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.582281 23016 raft_consensus.cc:2804] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:34.582649 23016 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:34.583047 23016 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.582782 22850 raft_consensus.cc:359] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.583489 22850 raft_consensus.cc:385] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:34.583771 22850 raft_consensus.cc:740] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:34.584239 23013 consensus_queue.cc:1048] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:34.584611 22850 consensus_queue.cc:260] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.587168 22850 ts_tablet_manager.cc:1434] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20251026 14:58:34.588133 22850 tablet_bootstrap.cc:492] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:34.589706 22971 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.594241 22850 tablet_bootstrap.cc:654] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:34.591418 22848 ts_tablet_manager.cc:1403] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.079s user 0.012s sys 0.005s
I20251026 14:58:34.591575 22849 tablet_bootstrap.cc:492] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:34.595352 22849 ts_tablet_manager.cc:1403] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.012s
I20251026 14:58:34.598517 23016 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.598817 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:34.600298 22731 raft_consensus.cc:3060] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.601613 23016 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:34.602766 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:34.603778 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:34.604482 22734 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.609680 22897 consensus_queue.cc:1048] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251026 14:58:34.612129 22734 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:34.613533 22624 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:34.614467 22897 raft_consensus.cc:2804] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:34.614953 22897 raft_consensus.cc:697] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:34.616371 22897 consensus_queue.cc:237] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.610555 22850 tablet_bootstrap.cc:492] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:34.604801 22898 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.617969 22850 ts_tablet_manager.cc:1403] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.005s
I20251026 14:58:34.618521 22806 raft_consensus.cc:2393] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 1: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:34.619911 22898 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:34.622320 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:34.623659 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:34.624389 22732 raft_consensus.cc:2393] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 1: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:58:34.625838 22773 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:34.627547 23021 consensus_queue.cc:1048] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:34.629482 23021 raft_consensus.cc:2749] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251026 14:58:34.637744 22971 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:34.639101 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:34.639415 22848 raft_consensus.cc:359] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.640075 22848 raft_consensus.cc:385] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:34.639686 22849 raft_consensus.cc:359] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.639732 22806 raft_consensus.cc:3060] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.640442 22849 raft_consensus.cc:385] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:34.640869 22849 raft_consensus.cc:740] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:34.640496 22850 raft_consensus.cc:359] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.641302 22850 raft_consensus.cc:385] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:34.641577 22850 raft_consensus.cc:740] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:34.641559 22849 consensus_queue.cc:260] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.642330 22850 consensus_queue.cc:260] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.643790 22849 ts_tablet_manager.cc:1434] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.048s user 0.002s sys 0.003s
I20251026 14:58:34.644635 22850 ts_tablet_manager.cc:1434] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20251026 14:58:34.645644 22850 tablet_bootstrap.cc:492] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:34.646046 22731 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:34.646729 22849 tablet_bootstrap.cc:492] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:34.647944 22806 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:34.649180 22622 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:34.653751 22850 tablet_bootstrap.cc:654] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:34.653949 22849 tablet_bootstrap.cc:654] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:34.656590 22971 raft_consensus.cc:2804] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:34.657310 22971 raft_consensus.cc:697] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:34.658089 22971 consensus_queue.cc:237] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.670883 22848 raft_consensus.cc:740] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:34.671626 22848 consensus_queue.cc:260] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.675727 22848 ts_tablet_manager.cc:1434] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.081s user 0.003s sys 0.002s
I20251026 14:58:34.677012 22848 tablet_bootstrap.cc:492] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:34.684419 22849 tablet_bootstrap.cc:492] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:34.686857 22849 ts_tablet_manager.cc:1403] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.040s user 0.017s sys 0.009s
I20251026 14:58:34.696128 22971 consensus_queue.cc:1048] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:34.702697 22555 catalog_manager.cc:5649] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 0 to 1, leader changed from <none> to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 1 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:34.707124 22555 catalog_manager.cc:5649] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 0 to 1, leader changed from <none> to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 1 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:34.707868 22849 raft_consensus.cc:359] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.706036 22850 tablet_bootstrap.cc:492] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:34.708873 22849 raft_consensus.cc:385] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:34.709157 22850 ts_tablet_manager.cc:1403] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.064s user 0.014s sys 0.003s
I20251026 14:58:34.709311 22849 raft_consensus.cc:740] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:34.713124 22848 tablet_bootstrap.cc:654] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:34.713712 22850 raft_consensus.cc:359] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.711506 22849 consensus_queue.cc:260] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.717413 22850 raft_consensus.cc:385] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:34.718122 22850 raft_consensus.cc:740] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:34.718447 22849 ts_tablet_manager.cc:1434] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.030s user 0.007s sys 0.000s
I20251026 14:58:34.718935 22850 consensus_queue.cc:260] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.719398 22849 tablet_bootstrap.cc:492] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:34.732234 22895 consensus_queue.cc:1048] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:34.745697 22850 ts_tablet_manager.cc:1434] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.036s user 0.004s sys 0.002s
I20251026 14:58:34.746767 22850 tablet_bootstrap.cc:492] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:34.750946 23021 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:34.751523 23021 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.756158 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:34.757618 22774 leader_election.cc:343] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:34.757985 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:34.757588 22849 tablet_bootstrap.cc:654] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:34.758608 22732 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:34.753764 23021 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:34.759974 22773 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:34.761018 23021 raft_consensus.cc:2804] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:34.761444 23021 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:34.761797 23021 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.765926 22848 tablet_bootstrap.cc:492] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:34.766443 22848 ts_tablet_manager.cc:1403] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.090s user 0.018s sys 0.009s
I20251026 14:58:34.769013 22848 raft_consensus.cc:359] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.770009 22848 raft_consensus.cc:385] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:34.770377 22848 raft_consensus.cc:740] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:34.771586 22848 consensus_queue.cc:260] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.774564 22848 ts_tablet_manager.cc:1434] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20251026 14:58:34.776655 22848 tablet_bootstrap.cc:492] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:34.777618 23021 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.780625 22850 tablet_bootstrap.cc:654] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:34.780581 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:34.781239 22731 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.782006 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:34.783998 23021 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:58:34.784409 22774 leader_election.cc:343] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:34.782646 22849 tablet_bootstrap.cc:492] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:34.785566 22849 ts_tablet_manager.cc:1403] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.066s user 0.007s sys 0.017s
I20251026 14:58:34.788307 22849 raft_consensus.cc:359] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.789099 22849 raft_consensus.cc:385] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:34.789458 22849 raft_consensus.cc:740] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:34.790035 22848 tablet_bootstrap.cc:654] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:34.790290 22849 consensus_queue.cc:260] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.814844 22731 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:34.819128 22773 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:34.820776 23021 raft_consensus.cc:2804] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:34.821450 23021 raft_consensus.cc:697] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:34.822074 22895 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:34.822515 23021 consensus_queue.cc:237] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.830308 22895 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.831669 22850 tablet_bootstrap.cc:492] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:34.832192 22850 ts_tablet_manager.cc:1403] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.086s user 0.011s sys 0.001s
I20251026 14:58:34.834558 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:34.834813 22850 raft_consensus.cc:359] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.835430 22850 raft_consensus.cc:385] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:34.835696 22850 raft_consensus.cc:740] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:34.835777 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:34.836367 22732 raft_consensus.cc:2468] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 0.
I20251026 14:58:34.836340 22850 consensus_queue.cc:260] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:34.838469 22622 leader_election.cc:343] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:34.841665 22624 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:34.842487 22971 raft_consensus.cc:2804] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:34.842862 22971 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:34.843010 22895 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:34.843202 22971 raft_consensus.cc:3060] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.843598 22937 consensus_queue.cc:1048] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:34.854543 22971 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.859079 22850 ts_tablet_manager.cc:1434] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.027s user 0.006s sys 0.002s
I20251026 14:58:34.859499 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:34.860105 22731 raft_consensus.cc:3060] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.861130 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:58:34.863125 22622 leader_election.cc:343] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:34.856714 22849 ts_tablet_manager.cc:1434] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.071s user 0.010s sys 0.024s
I20251026 14:58:34.864753 22849 tablet_bootstrap.cc:492] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:34.868059 22731 raft_consensus.cc:2468] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:34.861109 22971 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:34.869311 22624 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:34.871695 22848 tablet_bootstrap.cc:492] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:34.872258 22848 ts_tablet_manager.cc:1403] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.096s user 0.014s sys 0.006s
I20251026 14:58:34.873144 22850 tablet_bootstrap.cc:492] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:34.874825 22848 raft_consensus.cc:359] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.875496 22848 raft_consensus.cc:385] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:34.875748 22848 raft_consensus.cc:740] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:34.876329 22848 consensus_queue.cc:260] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.880672 22850 tablet_bootstrap.cc:654] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:34.881362 22896 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:34.881711 22849 tablet_bootstrap.cc:654] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:34.881878 22896 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.883989 22896 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:34.885723 22895 raft_consensus.cc:2804] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:34.886256 22895 raft_consensus.cc:697] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:34.887305 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:34.887051 22895 consensus_queue.cc:237] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.887943 22656 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:34.889000 22774 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:34.889896 23021 raft_consensus.cc:2804] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:34.890261 23021 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:34.890626 23021 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.892657 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:34.894513 22848 ts_tablet_manager.cc:1434] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.022s user 0.006s sys 0.001s
I20251026 14:58:34.895500 22848 tablet_bootstrap.cc:492] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
W20251026 14:58:34.896642 22773 leader_election.cc:343] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:34.899230 22896 raft_consensus.cc:493] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:34.899693 22896 raft_consensus.cc:515] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.899957 23021 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.901749 22896 leader_election.cc:290] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:34.902274 23031 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:34.902875 23021 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:34.902743 23031 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.904613 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:34.905408 22656 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.905500 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:34.906248 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:34.906847 22805 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:34.888832 22555 catalog_manager.cc:5649] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
W20251026 14:58:34.909580 22774 leader_election.cc:343] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:34.904853 23031 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:34.911717 22697 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:34.912792 22937 consensus_queue.cc:1048] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:34.913441 22656 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:34.913545 23031 raft_consensus.cc:2804] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:34.913990 23031 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:34.914363 23031 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.914645 22937 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:34.915102 22937 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.917205 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:34.917124 22937 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:34.919427 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:34.919600 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:34.920068 22732 raft_consensus.cc:2468] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:34.920233 22806 raft_consensus.cc:2468] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:34.921439 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:34.923328 22774 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:34.924490 22697 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:34.921461 23031 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:34.925135 22700 leader_election.cc:343] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:34.925550 22991 raft_consensus.cc:2804] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:34.925683 23021 raft_consensus.cc:2804] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:34.926033 22991 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:34.926433 22991 raft_consensus.cc:3060] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.927218 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:34.927692 23021 raft_consensus.cc:697] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:34.927851 22659 raft_consensus.cc:2468] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:34.928563 23021 consensus_queue.cc:237] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.929225 22773 leader_election.cc:304] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:34.930675 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:34.931263 22806 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.931869 22898 raft_consensus.cc:2804] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:34.932236 22898 raft_consensus.cc:493] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:58:34.932545 22773 leader_election.cc:343] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:34.932569 22898 raft_consensus.cc:3060] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.937544 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:34.938874 22806 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
W20251026 14:58:34.940860 22700 leader_election.cc:343] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:34.941817 22697 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:34.942616 22937 raft_consensus.cc:2804] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:34.942718 23031 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:34.943264 22937 raft_consensus.cc:697] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:34.944713 22937 consensus_queue.cc:237] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.945560 22849 tablet_bootstrap.cc:492] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:34.946030 22849 ts_tablet_manager.cc:1403] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.082s user 0.013s sys 0.000s
I20251026 14:58:34.954969 22848 tablet_bootstrap.cc:654] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:34.960312 23032 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:34.960866 23032 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.962581 22849 raft_consensus.cc:359] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.963048 23032 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:34.963301 22849 raft_consensus.cc:385] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:34.963619 22849 raft_consensus.cc:740] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:34.964298 22849 consensus_queue.cc:260] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.966571 22849 ts_tablet_manager.cc:1434] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20251026 14:58:34.966565 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:34.967309 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:34.967437 22805 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:34.968890 22697 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:34.970011 23031 raft_consensus.cc:2804] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251026 14:58:34.970327 22700 leader_election.cc:343] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:34.970536 23031 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:34.971027 23031 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.972597 22849 tablet_bootstrap.cc:492] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:34.979086 22848 tablet_bootstrap.cc:492] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:34.981937 22848 ts_tablet_manager.cc:1403] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.087s user 0.010s sys 0.003s
I20251026 14:58:34.981513 22898 raft_consensus.cc:515] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.985481 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:34.985538 22898 leader_election.cc:290] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:34.989066 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:34.987080 22991 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:34.993767 22731 raft_consensus.cc:3060] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:34.999603 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:35.000262 22659 raft_consensus.cc:3060] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.003180 22849 tablet_bootstrap.cc:654] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.004817 22991 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:35.013207 22850 tablet_bootstrap.cc:492] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:35.014803 22850 ts_tablet_manager.cc:1403] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.142s user 0.043s sys 0.029s
I20251026 14:58:35.017599 22850 raft_consensus.cc:359] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.018306 22850 raft_consensus.cc:385] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.018581 22850 raft_consensus.cc:740] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.009434 22556 catalog_manager.cc:5649] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:35.019208 22850 consensus_queue.cc:260] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.021971 22850 ts_tablet_manager.cc:1434] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20251026 14:58:35.023463 22850 tablet_bootstrap.cc:492] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:35.029584 22850 tablet_bootstrap.cc:654] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.035112 22850 tablet_bootstrap.cc:492] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:35.035590 22850 ts_tablet_manager.cc:1403] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20251026 14:58:35.036209 22731 raft_consensus.cc:2468] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:35.036904 22849 tablet_bootstrap.cc:492] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:35.037451 22849 ts_tablet_manager.cc:1403] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.065s user 0.013s sys 0.008s
I20251026 14:58:35.038136 22850 raft_consensus.cc:359] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.038843 22850 raft_consensus.cc:385] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.039117 22850 raft_consensus.cc:740] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:34.989758 22848 raft_consensus.cc:359] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.039985 22850 consensus_queue.cc:260] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:34.988262 22774 leader_election.cc:343] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.040212 22849 raft_consensus.cc:359] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:35.040968 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 52782.6us
I20251026 14:58:35.041304 22849 raft_consensus.cc:385] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:34.992172 23031 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.041882 22849 raft_consensus.cc:740] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.042516 22850 ts_tablet_manager.cc:1434] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20251026 14:58:35.042698 22849 consensus_queue.cc:260] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.043493 22850 tablet_bootstrap.cc:492] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:35.043383 23031 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:35.043818 22898 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:35.044308 22898 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.045600 22849 ts_tablet_manager.cc:1434] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20251026 14:58:35.046108 22898 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:35.046625 22849 tablet_bootstrap.cc:492] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:35.046715 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:35.047319 22807 raft_consensus.cc:3060] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.049834 22850 tablet_bootstrap.cc:654] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.045113 22773 leader_election.cc:304] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:35.053598 22849 tablet_bootstrap.cc:654] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.054328 22898 raft_consensus.cc:2804] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:35.054462 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:35.055096 22898 raft_consensus.cc:697] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:35.055948 22898 consensus_queue.cc:237] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.056360 22555 catalog_manager.cc:5649] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 0 to 1, leader changed from <none> to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:35.063865 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:35.071105 22971 consensus_queue.cc:1048] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:35.072407 22896 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:35.073069 22896 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.075685 22896 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:35.065923 22554 catalog_manager.cc:5649] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 0 to 1, leader changed from <none> to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 1 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:35.079345 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:35.080227 22804 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.083935 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:35.085255 22991 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:35.085762 22991 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.087824 22991 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:58:35.088517 22774 leader_election.cc:343] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
W20251026 14:58:35.089159 22604 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251026 14:58:35.084110 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:35.090376 22731 raft_consensus.cc:2468] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
W20251026 14:58:35.097153 22700 leader_election.cc:343] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.097330 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:35.098162 22731 raft_consensus.cc:2393] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 1: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:58:35.103502 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:35.104950 22773 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:35.105713 22896 raft_consensus.cc:2804] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:35.106094 22896 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:35.106437 22896 raft_consensus.cc:3060] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.107606 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:58:35.109439 22697 leader_election.cc:343] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.109891 22850 tablet_bootstrap.cc:492] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:35.110380 22850 ts_tablet_manager.cc:1403] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.067s user 0.035s sys 0.016s
I20251026 14:58:35.113194 22850 raft_consensus.cc:359] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.114017 22850 raft_consensus.cc:385] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.114404 22850 raft_consensus.cc:740] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.114576 22937 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:35.115028 22850 consensus_queue.cc:260] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.115053 22937 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.117300 22937 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:35.117640 22850 ts_tablet_manager.cc:1434] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20251026 14:58:35.117843 22895 consensus_queue.cc:1048] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:35.118548 22850 tablet_bootstrap.cc:492] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:35.119410 22773 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:58:35.114144 22700 leader_election.cc:343] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.120571 22700 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:35.121384 22937 raft_consensus.cc:2749] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:35.123428 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:35.123517 22896 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.124280 22656 raft_consensus.cc:2393] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 1: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:35.125768 22896 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:35.125870 22850 tablet_bootstrap.cc:654] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.126425 22896 raft_consensus.cc:3060] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.133123 22849 tablet_bootstrap.cc:492] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:35.133666 22849 ts_tablet_manager.cc:1403] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.087s user 0.029s sys 0.002s
I20251026 14:58:35.133666 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:35.136116 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:35.137032 22849 raft_consensus.cc:359] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.137770 22849 raft_consensus.cc:385] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.138098 22849 raft_consensus.cc:740] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.138872 22849 consensus_queue.cc:260] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:35.139859 22774 leader_election.cc:343] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.142066 22896 raft_consensus.cc:2749] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:35.143168 22807 raft_consensus.cc:2468] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:35.139261 22804 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:35.144524 22700 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:35.040341 22848 raft_consensus.cc:385] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.148087 22848 raft_consensus.cc:740] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.145253 22659 raft_consensus.cc:2468] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:35.151604 22849 ts_tablet_manager.cc:1434] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.017s user 0.005s sys 0.003s
I20251026 14:58:35.152755 22849 tablet_bootstrap.cc:492] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:35.154059 22697 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:35.155997 23031 raft_consensus.cc:2804] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:35.156635 23031 raft_consensus.cc:697] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:35.157972 23031 consensus_queue.cc:237] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.160094 22850 tablet_bootstrap.cc:492] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:35.160414 22937 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.160616 22850 ts_tablet_manager.cc:1403] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.042s user 0.025s sys 0.014s
I20251026 14:58:35.160862 22849 tablet_bootstrap.cc:654] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.161085 22697 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:35.162575 23032 raft_consensus.cc:2804] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:35.165151 22850 raft_consensus.cc:359] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.166250 22850 raft_consensus.cc:385] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.166663 22850 raft_consensus.cc:740] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.167544 22850 consensus_queue.cc:260] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.171394 22850 ts_tablet_manager.cc:1434] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20251026 14:58:35.172551 22849 tablet_bootstrap.cc:492] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:35.174116 22850 tablet_bootstrap.cc:492] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:35.174151 22849 ts_tablet_manager.cc:1403] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.003s
I20251026 14:58:35.178658 22850 tablet_bootstrap.cc:654] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.179312 22848 consensus_queue.cc:260] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.211608 22937 raft_consensus.cc:2749] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:35.180153 22849 raft_consensus.cc:359] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.215492 22849 raft_consensus.cc:385] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.215014 22971 consensus_queue.cc:1048] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:35.215859 22849 raft_consensus.cc:740] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.216686 22849 consensus_queue.cc:260] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.220430 22849 ts_tablet_manager.cc:1434] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.046s user 0.007s sys 0.000s
I20251026 14:58:35.221100 22878 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:35.224251 22848 ts_tablet_manager.cc:1434] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.242s user 0.006s sys 0.001s
I20251026 14:58:35.243575 22878 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.244282 22848 tablet_bootstrap.cc:492] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:35.245262 22878 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:35.228323 23032 raft_consensus.cc:697] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:35.235234 23031 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:35.247159 23031 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.246740 23032 consensus_queue.cc:237] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.248582 22849 tablet_bootstrap.cc:492] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:35.249156 23031 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:35.253778 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:58:35.236779 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 117483us
I20251026 14:58:35.239835 22971 consensus_queue.cc:1048] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20251026 14:58:35.255892 22697 leader_election.cc:343] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.256361 22878 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:35.256901 22878 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.260202 22878 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:35.269212 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:35.269533 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:35.271958 22774 leader_election.cc:343] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
W20251026 14:58:35.272941 22774 leader_election.cc:343] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.275564 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:35.277693 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:35.278278 22731 raft_consensus.cc:3060] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.284624 22848 tablet_bootstrap.cc:654] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.286460 22731 raft_consensus.cc:2468] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:35.289222 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:35.290027 22732 raft_consensus.cc:2468] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:35.291236 22555 catalog_manager.cc:5649] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:35.294404 22849 tablet_bootstrap.cc:654] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.295369 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:35.301782 22732 raft_consensus.cc:2468] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:35.296669 22848 tablet_bootstrap.cc:492] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:35.305456 22848 ts_tablet_manager.cc:1403] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.061s user 0.012s sys 0.002s
I20251026 14:58:35.305104 22556 catalog_manager.cc:5649] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 0 to 1, leader changed from <none> to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:35.308727 22848 raft_consensus.cc:359] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.309355 22848 raft_consensus.cc:385] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.306843 22556 catalog_manager.cc:5649] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 0 to 1, leader changed from <none> to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:35.309644 22848 raft_consensus.cc:740] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.297778 22850 tablet_bootstrap.cc:492] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:35.298633 22773 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:35.311156 22898 raft_consensus.cc:2804] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:35.311461 23076 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:35.311632 22773 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:35.311985 23076 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.312481 23021 raft_consensus.cc:2804] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:35.312856 23021 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:35.313217 23021 raft_consensus.cc:3060] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.314172 23076 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:58:35.300525 22700 leader_election.cc:343] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.314878 22700 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:35.315060 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:35.311643 22898 raft_consensus.cc:697] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:35.316202 23032 raft_consensus.cc:2749] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:35.316501 22898 consensus_queue.cc:237] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.318643 22850 ts_tablet_manager.cc:1403] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.145s user 0.025s sys 0.058s
I20251026 14:58:35.318735 22895 consensus_queue.cc:1048] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 14:58:35.320361 22774 leader_election.cc:343] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.322741 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:35.323344 22773 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:35.322121 22850 raft_consensus.cc:359] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.324034 22850 raft_consensus.cc:385] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.324326 22850 raft_consensus.cc:740] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.325510 23076 raft_consensus.cc:2804] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:35.325894 23076 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:35.326241 23076 raft_consensus.cc:3060] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
W20251026 14:58:35.329558 22773 leader_election.cc:343] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.328996 22848 consensus_queue.cc:260] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.329996 22773 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:35.331517 22896 raft_consensus.cc:2749] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:35.332096 22878 consensus_queue.cc:1048] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:35.332595 22848 ts_tablet_manager.cc:1434] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20251026 14:58:35.334494 22850 consensus_queue.cc:260] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.337306 22850 ts_tablet_manager.cc:1434] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.018s user 0.004s sys 0.001s
I20251026 14:58:35.338445 22850 tablet_bootstrap.cc:492] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:35.339331 23021 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.340451 22849 tablet_bootstrap.cc:492] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:35.341442 22848 tablet_bootstrap.cc:492] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:35.347606 22850 tablet_bootstrap.cc:654] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.348091 22849 ts_tablet_manager.cc:1403] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.105s user 0.015s sys 0.005s
I20251026 14:58:35.351014 22895 consensus_queue.cc:1048] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:35.355494 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:58:35.357576 22774 leader_election.cc:343] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.359210 23021 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:35.370093 22849 raft_consensus.cc:359] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.375474 22849 raft_consensus.cc:385] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.376078 22849 raft_consensus.cc:740] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.403112 22849 consensus_queue.cc:260] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.377516 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:35.411717 22732 raft_consensus.cc:3060] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.393623 22898 consensus_queue.cc:1048] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:35.394605 23076 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.400475 22659 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.400936 22896 consensus_queue.cc:1048] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:35.424319 23076 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:35.371711 22848 tablet_bootstrap.cc:654] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.430158 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:35.420921 22732 raft_consensus.cc:2468] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
W20251026 14:58:35.433907 22774 leader_election.cc:343] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.433185 22849 ts_tablet_manager.cc:1434] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.084s user 0.003s sys 0.004s
I20251026 14:58:35.435734 22849 tablet_bootstrap.cc:492] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:35.409973 22556 catalog_manager.cc:5649] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:35.436324 23076 consensus_queue.cc:1048] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:35.429780 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:35.450526 22731 raft_consensus.cc:3060] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.450994 22849 tablet_bootstrap.cc:654] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.441854 22850 tablet_bootstrap.cc:492] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:35.454041 22850 ts_tablet_manager.cc:1403] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.116s user 0.037s sys 0.043s
I20251026 14:58:35.456681 22850 raft_consensus.cc:359] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.466096 22850 raft_consensus.cc:385] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.466454 22850 raft_consensus.cc:740] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.459200 22773 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:35.469810 22898 raft_consensus.cc:2804] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:35.473664 22898 raft_consensus.cc:697] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:35.474630 22898 consensus_queue.cc:237] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.467118 22850 consensus_queue.cc:260] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.478608 23080 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:35.479108 23080 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.481715 22850 ts_tablet_manager.cc:1434] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.027s user 0.001s sys 0.005s
I20251026 14:58:35.482678 23080 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:35.484059 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:35.464614 22731 raft_consensus.cc:2468] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
W20251026 14:58:35.485725 22700 leader_election.cc:343] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.489857 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:35.491811 22937 consensus_queue.cc:1048] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 14:58:35.493964 22697 leader_election.cc:343] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.494339 22773 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:35.495589 23076 raft_consensus.cc:2804] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:35.496104 23076 raft_consensus.cc:697] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:35.496930 23076 consensus_queue.cc:237] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.498934 22848 tablet_bootstrap.cc:492] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:35.499527 22848 ts_tablet_manager.cc:1403] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.158s user 0.012s sys 0.000s
I20251026 14:58:35.494468 22697 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:35.502031 22937 raft_consensus.cc:2749] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:35.502660 22848 raft_consensus.cc:359] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.503545 22937 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:35.503669 22848 raft_consensus.cc:385] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.504174 22848 raft_consensus.cc:740] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.504070 22937 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.505141 22848 consensus_queue.cc:260] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.507091 22848 ts_tablet_manager.cc:1434] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20251026 14:58:35.507151 22937 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:35.508327 22848 tablet_bootstrap.cc:492] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:35.509403 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:35.510665 22700 leader_election.cc:343] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.515829 22656 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.522730 22850 tablet_bootstrap.cc:492] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:35.523554 22849 tablet_bootstrap.cc:492] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:35.524137 22849 ts_tablet_manager.cc:1403] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.089s user 0.017s sys 0.009s
I20251026 14:58:35.526875 22849 raft_consensus.cc:359] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.527196 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:35.533726 22850 tablet_bootstrap.cc:654] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.533170 22849 raft_consensus.cc:385] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.534399 22849 raft_consensus.cc:740] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
W20251026 14:58:35.534436 22697 leader_election.cc:343] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.536015 22697 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:35.536849 23032 raft_consensus.cc:2749] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:35.538460 22848 tablet_bootstrap.cc:654] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.536374 22849 consensus_queue.cc:260] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.545105 22849 ts_tablet_manager.cc:1434] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.021s user 0.005s sys 0.001s
I20251026 14:58:35.545956 22971 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:35.546530 22849 tablet_bootstrap.cc:492] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:35.546456 22971 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.547847 23032 consensus_queue.cc:1048] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:35.549206 22554 catalog_manager.cc:5649] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:35.551319 22554 catalog_manager.cc:5649] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:35.555032 22971 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:35.555032 22849 tablet_bootstrap.cc:654] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.555025 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:35.558184 22848 tablet_bootstrap.cc:492] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:35.558964 22848 ts_tablet_manager.cc:1403] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.051s user 0.013s sys 0.001s
I20251026 14:58:35.560954 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:35.562556 22848 raft_consensus.cc:359] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.565313 22624 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:35.568231 22848 raft_consensus.cc:385] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.568775 22848 raft_consensus.cc:740] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
W20251026 14:58:35.570633 22774 consensus_peers.cc:597] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 -> Peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Couldn't send request to peer df3a0d1619ee48e8ab68ee0543a72a29. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:35.570875 22848 consensus_queue.cc:260] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.581264 22848 ts_tablet_manager.cc:1434] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.022s user 0.006s sys 0.001s
I20251026 14:58:35.582086 22849 tablet_bootstrap.cc:492] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:35.582434 22850 tablet_bootstrap.cc:492] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:35.582578 22849 ts_tablet_manager.cc:1403] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.036s user 0.011s sys 0.000s
I20251026 14:58:35.582887 22850 ts_tablet_manager.cc:1403] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.060s user 0.015s sys 0.028s
I20251026 14:58:35.585485 22850 raft_consensus.cc:359] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.586158 22850 raft_consensus.cc:385] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.586438 22850 raft_consensus.cc:740] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.587059 22848 tablet_bootstrap.cc:492] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:35.587705 22849 raft_consensus.cc:359] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.587075 22850 consensus_queue.cc:260] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.588333 22849 raft_consensus.cc:385] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.588634 22849 raft_consensus.cc:740] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.589268 22849 consensus_queue.cc:260] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.591676 22849 ts_tablet_manager.cc:1434] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20251026 14:58:35.592232 22850 ts_tablet_manager.cc:1434] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20251026 14:58:35.592629 22849 tablet_bootstrap.cc:492] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:35.605458 22897 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:35.606001 22897 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.606984 22849 tablet_bootstrap.cc:654] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.607924 22897 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:35.609942 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:35.596678 22850 tablet_bootstrap.cc:492] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:35.597643 22848 tablet_bootstrap.cc:654] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.599179 22971 raft_consensus.cc:2749] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:35.610626 22805 raft_consensus.cc:2468] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 0.
I20251026 14:58:35.613335 22622 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:35.617859 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:35.618654 22850 tablet_bootstrap.cc:654] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
W20251026 14:58:35.619809 22624 leader_election.cc:343] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.621009 22937 consensus_queue.cc:1048] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:35.626864 22971 raft_consensus.cc:2804] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:35.627326 22971 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:35.627667 22971 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.644358 22971 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.646562 22971 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:35.653307 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:35.653985 22805 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.654227 22937 consensus_queue.cc:1048] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:35.656814 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:58:35.667110 22624 leader_election.cc:343] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.671461 22848 tablet_bootstrap.cc:492] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:35.671965 22848 ts_tablet_manager.cc:1403] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.085s user 0.003s sys 0.018s
I20251026 14:58:35.663854 22805 raft_consensus.cc:2468] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:35.677706 22622 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:35.678704 22897 raft_consensus.cc:2804] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:35.678982 22848 raft_consensus.cc:359] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.679392 22897 raft_consensus.cc:697] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:35.679745 22848 raft_consensus.cc:385] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.680070 22848 raft_consensus.cc:740] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.680783 22848 consensus_queue.cc:260] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.680563 22897 consensus_queue.cc:237] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.683507 22848 ts_tablet_manager.cc:1434] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20251026 14:58:35.684581 22848 tablet_bootstrap.cc:492] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:35.686126 22849 tablet_bootstrap.cc:492] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:35.686775 22849 ts_tablet_manager.cc:1403] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.094s user 0.019s sys 0.002s
I20251026 14:58:35.687222 22898 consensus_queue.cc:1048] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:35.690768 22848 tablet_bootstrap.cc:654] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.717787 22849 raft_consensus.cc:359] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.718851 22849 raft_consensus.cc:385] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.719215 22848 tablet_bootstrap.cc:492] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:35.719715 22848 ts_tablet_manager.cc:1403] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.003s
I20251026 14:58:35.719214 22849 raft_consensus.cc:740] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.722908 22850 tablet_bootstrap.cc:492] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:35.722525 22848 raft_consensus.cc:359] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.723420 22850 ts_tablet_manager.cc:1403] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.127s user 0.041s sys 0.067s
I20251026 14:58:35.723506 22848 raft_consensus.cc:385] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.721040 22849 consensus_queue.cc:260] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.723862 22848 raft_consensus.cc:740] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.724519 22848 consensus_queue.cc:260] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.725656 22849 ts_tablet_manager.cc:1434] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.039s user 0.008s sys 0.002s
I20251026 14:58:35.726140 22850 raft_consensus.cc:359] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.726789 22850 raft_consensus.cc:385] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.726984 22848 ts_tablet_manager.cc:1434] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20251026 14:58:35.727083 22850 raft_consensus.cc:740] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.727790 22849 tablet_bootstrap.cc:492] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:35.728024 22848 tablet_bootstrap.cc:492] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:35.727793 22850 consensus_queue.cc:260] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.734077 22849 tablet_bootstrap.cc:654] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.735304 22850 ts_tablet_manager.cc:1434] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20251026 14:58:35.735502 22848 tablet_bootstrap.cc:654] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.738050 22898 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:35.739579 22898 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.743275 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:35.743470 22850 tablet_bootstrap.cc:492] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
W20251026 14:58:35.744688 22774 leader_election.cc:343] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.745886 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:35.746600 22731 raft_consensus.cc:2468] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:35.755605 22850 tablet_bootstrap.cc:654] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.743494 22898 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:35.762154 22773 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:35.767814 23076 raft_consensus.cc:2804] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:35.768388 23076 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:35.768798 23076 raft_consensus.cc:3060] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.777209 23032 consensus_queue.cc:1048] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:35.778919 23078 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:35.779436 23078 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.782009 23076 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.783138 23078 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:35.785193 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:35.786150 22971 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:35.786655 22971 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:35.787765 22774 leader_election.cc:343] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.788995 22971 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:35.791019 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:35.806735 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:58:35.808100 22774 leader_election.cc:343] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.810595 23076 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:35.816187 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:35.816911 22731 raft_consensus.cc:2468] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:35.824054 22898 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:35.825048 22898 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.825841 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:35.826426 22731 raft_consensus.cc:3060] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.828900 22898 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:35.834640 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:35.838208 22552 catalog_manager.cc:5649] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 0 to 1, leader changed from <none> to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 1 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
W20251026 14:58:35.851379 22774 leader_election.cc:343] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.855315 22971 raft_consensus.cc:493] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:35.856117 22971 raft_consensus.cc:515] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.858899 22971 leader_election.cc:290] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:35.862215 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:35.870127 22850 tablet_bootstrap.cc:492] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:35.870651 22850 ts_tablet_manager.cc:1403] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.127s user 0.043s sys 0.065s
I20251026 14:58:35.873353 22850 raft_consensus.cc:359] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.874063 22850 raft_consensus.cc:385] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.874356 22850 raft_consensus.cc:740] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.874920 22850 consensus_queue.cc:260] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.876819 22773 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:35.877293 22850 ts_tablet_manager.cc:1434] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251026 14:58:35.878453 22849 tablet_bootstrap.cc:492] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:35.878880 22849 ts_tablet_manager.cc:1403] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.151s user 0.037s sys 0.064s
I20251026 14:58:35.878971 22898 raft_consensus.cc:2804] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:35.879364 22898 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:35.879700 22898 raft_consensus.cc:3060] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.880553 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:35.881345 22734 raft_consensus.cc:2393] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 1: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:35.882193 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:35.885185 22656 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.884958 22849 raft_consensus.cc:359] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.885807 22849 raft_consensus.cc:385] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.886080 22849 raft_consensus.cc:740] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.886785 22849 consensus_queue.cc:260] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.888803 22849 ts_tablet_manager.cc:1434] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20251026 14:58:35.889676 22849 tablet_bootstrap.cc:492] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:35.890554 22850 tablet_bootstrap.cc:492] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:35.890591 22624 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:35.893198 22624 leader_election.cc:304] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:35.893985 22895 raft_consensus.cc:3060] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.895625 22849 tablet_bootstrap.cc:654] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.899066 22850 tablet_bootstrap.cc:654] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.905500 22897 raft_consensus.cc:2749] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:35.905838 23078 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:35.906374 23078 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.906766 22849 tablet_bootstrap.cc:492] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:35.907398 22849 ts_tablet_manager.cc:1403] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20251026 14:58:35.908456 23078 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:35.909927 22849 raft_consensus.cc:359] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.910610 22849 raft_consensus.cc:385] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.910887 22849 raft_consensus.cc:740] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.911523 22849 consensus_queue.cc:260] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.915176 22850 tablet_bootstrap.cc:492] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:35.915324 23080 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:35.915747 22850 ts_tablet_manager.cc:1403] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.025s user 0.019s sys 0.000s
I20251026 14:58:35.915815 23080 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.916875 22898 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.917879 23080 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:35.918350 22850 raft_consensus.cc:359] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.918998 22850 raft_consensus.cc:385] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.919274 22850 raft_consensus.cc:740] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.919493 22898 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:35.891494 22848 tablet_bootstrap.cc:492] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:35.919860 22850 consensus_queue.cc:260] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.921450 22848 ts_tablet_manager.cc:1403] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.194s user 0.016s sys 0.017s
I20251026 14:58:35.925503 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:35.921694 22895 raft_consensus.cc:2749] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:35.922212 22849 ts_tablet_manager.cc:1434] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20251026 14:58:35.926028 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:58:35.926385 22774 consensus_peers.cc:597] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 -> Peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Couldn't send request to peer df3a0d1619ee48e8ab68ee0543a72a29. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:35.926627 22732 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:35.874172 22731 raft_consensus.cc:2468] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:35.927124 22849 tablet_bootstrap.cc:492] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:35.922173 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:35.923355 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:35.929466 22804 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:35.923597 22850 ts_tablet_manager.cc:1434] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20251026 14:58:35.930351 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:35.930914 22697 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:35.931216 22850 tablet_bootstrap.cc:492] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:35.931952 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:35.932571 22729 raft_consensus.cc:3060] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.931782 23080 raft_consensus.cc:2804] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:35.933975 22849 tablet_bootstrap.cc:654] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.935452 23080 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:35.935853 23080 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.936843 22848 raft_consensus.cc:359] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.938023 22848 raft_consensus.cc:385] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:35.938906 22848 raft_consensus.cc:740] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:35.943456 22848 consensus_queue.cc:260] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.945729 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:35.947405 22848 ts_tablet_manager.cc:1434] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
W20251026 14:58:35.941978 22774 consensus_peers.cc:597] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 -> Peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Couldn't send request to peer df3a0d1619ee48e8ab68ee0543a72a29. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20251026 14:58:35.950795 22700 leader_election.cc:343] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
W20251026 14:58:35.952346 22774 leader_election.cc:343] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
W20251026 14:58:35.952984 22774 leader_election.cc:343] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.956705 23076 consensus_queue.cc:1048] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:35.957768 22658 raft_consensus.cc:3060] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.940586 22729 raft_consensus.cc:2468] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:35.948594 22773 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:35.950852 23032 consensus_queue.cc:1048] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20251026 14:58:35.951642 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 53769.5us
I20251026 14:58:35.959568 22773 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:35.960373 22898 raft_consensus.cc:2804] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:35.960719 22898 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:35.961146 22896 raft_consensus.cc:2804] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
W20251026 14:58:35.961239 22773 leader_election.cc:343] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:35.961740 22896 raft_consensus.cc:697] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:35.961899 22773 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:35.961043 22898 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:35.954572 22848 tablet_bootstrap.cc:492] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:35.962596 22896 consensus_queue.cc:237] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.965454 23021 raft_consensus.cc:2749] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:35.948391 22850 tablet_bootstrap.cc:654] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:35.968943 22898 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:35.980894 22898 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:35.988174 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:35.991204 22849 tablet_bootstrap.cc:492] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:35.991906 22849 ts_tablet_manager.cc:1403] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.065s user 0.007s sys 0.005s
I20251026 14:58:35.999547 22773 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:35.999974 22850 tablet_bootstrap.cc:492] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:36.001554 22850 ts_tablet_manager.cc:1403] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.071s user 0.023s sys 0.009s
I20251026 14:58:36.006522 23032 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:36.007231 22850 raft_consensus.cc:359] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.008306 22850 raft_consensus.cc:385] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:36.008742 22850 raft_consensus.cc:740] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:36.010094 22850 consensus_queue.cc:260] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.011567 22896 raft_consensus.cc:2804] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:36.013485 22896 raft_consensus.cc:697] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
W20251026 14:58:36.014145 22774 leader_election.cc:343] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:36.014392 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:36.007050 23032 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.014744 22896 consensus_queue.cc:237] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.021611 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:36.007047 23080 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.017249 23032 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:36.069042 22937 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:36.069832 22937 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.107023 22731 raft_consensus.cc:2393] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 1: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:36.108263 22848 tablet_bootstrap.cc:654] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:36.133078 22937 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:36.097244 23126 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:36.145354 23126 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.146427 22849 raft_consensus.cc:359] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.147137 22849 raft_consensus.cc:385] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:36.147418 22849 raft_consensus.cc:740] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:36.148020 22849 consensus_queue.cc:260] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:36.151772 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 110277us
I20251026 14:58:36.155117 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:36.157689 22700 leader_election.cc:343] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
W20251026 14:58:36.137476 23130 log.cc:927] Time spent T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.114s user 0.004s sys 0.009s
I20251026 14:58:36.137830 23080 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:36.167331 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:36.168453 22850 ts_tablet_manager.cc:1434] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.167s user 0.042s sys 0.077s
I20251026 14:58:36.169200 22850 tablet_bootstrap.cc:492] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:36.173746 22804 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:36.179569 22849 ts_tablet_manager.cc:1434] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.187s user 0.005s sys 0.009s
I20251026 14:58:36.180747 22849 tablet_bootstrap.cc:492] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:36.184213 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:36.170231 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:36.200096 23126 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:36.199525 22552 catalog_manager.cc:5649] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:36.203330 22897 consensus_queue.cc:1048] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 14:58:36.214421 22700 leader_election.cc:343] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
W20251026 14:58:36.215097 22700 leader_election.cc:343] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:36.214125 22878 consensus_queue.cc:1048] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20251026 14:58:36.220328 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:36.222152 22774 leader_election.cc:343] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:36.228019 22848 tablet_bootstrap.cc:492] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:36.228574 22848 ts_tablet_manager.cc:1403] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.279s user 0.023s sys 0.001s
I20251026 14:58:36.225703 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:36.238854 22806 raft_consensus.cc:2393] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 1: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:36.223207 23126 consensus_queue.cc:1048] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20251026 14:58:36.248023 22850 tablet_bootstrap.cc:654] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:36.248535 23136 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:36.226074 22697 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:36.227144 22773 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:36.251704 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:36.251624 23136 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:36.253835 22697 leader_election.cc:343] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:36.254257 22728 raft_consensus.cc:2468] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:36.254423 22991 raft_consensus.cc:2804] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:36.254776 22991 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:36.255162 22991 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:36.255246 23136 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:36.264101 22878 raft_consensus.cc:2749] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251026 14:58:36.267107 22848 raft_consensus.cc:359] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.267665 22848 raft_consensus.cc:385] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:36.267863 22848 raft_consensus.cc:740] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:36.254261 22697 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:58:36.269740 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 47613.1us
I20251026 14:58:36.270058 22697 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:36.270421 22849 tablet_bootstrap.cc:654] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:36.268368 22848 consensus_queue.cc:260] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.271428 23136 raft_consensus.cc:2749] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251026 14:58:36.270839 23021 consensus_queue.cc:1048] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251026 14:58:36.271469 23132 raft_consensus.cc:2749] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:36.280558 22848 ts_tablet_manager.cc:1434] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.052s user 0.003s sys 0.002s
I20251026 14:58:36.280447 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:36.281359 22848 tablet_bootstrap.cc:492] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
W20251026 14:58:36.282238 22700 leader_election.cc:343] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:36.284994 23131 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:36.286545 23131 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.288604 23131 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:36.293679 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:58:36.296227 22624 consensus_peers.cc:597] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 -> Peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Couldn't send request to peer 562bb1e1c8e445b8baecd29c3669658e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:36.301916 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:36.302135 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:36.303126 22806 raft_consensus.cc:2468] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:36.309126 22555 catalog_manager.cc:5649] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:36.312023 22773 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:36.313231 23126 raft_consensus.cc:2804] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:36.314391 23126 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:36.316380 22730 raft_consensus.cc:2393] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 1: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:36.316706 23126 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:36.317757 22697 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:36.316264 22991 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.329667 23021 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:36.333768 22624 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:36.335320 22895 raft_consensus.cc:3060] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:36.341862 22850 tablet_bootstrap.cc:492] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:36.342489 22850 ts_tablet_manager.cc:1403] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.173s user 0.016s sys 0.003s
I20251026 14:58:36.343852 23136 raft_consensus.cc:2804] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:36.344270 23136 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:36.344604 23136 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:36.330155 23021 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.369061 23021 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:36.369475 22849 tablet_bootstrap.cc:492] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:36.369956 22849 ts_tablet_manager.cc:1403] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.189s user 0.008s sys 0.012s
I20251026 14:58:36.375944 22848 tablet_bootstrap.cc:654] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:36.376219 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:36.388690 22849 raft_consensus.cc:359] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.389755 22849 raft_consensus.cc:385] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:36.390057 22849 raft_consensus.cc:740] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:36.391778 22849 consensus_queue.cc:260] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.393710 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:36.383407 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:36.398731 22850 raft_consensus.cc:359] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.399344 22850 raft_consensus.cc:385] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:36.399680 22850 raft_consensus.cc:740] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
W20251026 14:58:36.405241 22700 leader_election.cc:343] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:36.406029 22850 consensus_queue.cc:260] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.392428 22991 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:58:36.413798 22774 leader_election.cc:343] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:36.398989 22729 raft_consensus.cc:2468] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:36.415292 22850 ts_tablet_manager.cc:1434] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.073s user 0.004s sys 0.003s
I20251026 14:58:36.416785 22850 tablet_bootstrap.cc:492] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:36.416333 22849 ts_tablet_manager.cc:1434] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.046s user 0.005s sys 0.003s
I20251026 14:58:36.419577 22849 tablet_bootstrap.cc:492] T a673ec034b704e8e86556afe9322d8c2 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:36.414791 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:36.421736 22896 consensus_queue.cc:1048] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20251026 14:58:36.436844 23136 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.430759 22806 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:36.452890 23136 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:36.454425 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:36.440225 22991 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:58:36.456871 22700 leader_election.cc:343] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:36.456163 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:36.457165 22895 raft_consensus.cc:2749] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:36.458963 22773 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:36.469928 23078 raft_consensus.cc:2804] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:36.454466 23126 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.472254 23078 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:36.472621 23078 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
W20251026 14:58:36.476295 22774 consensus_peers.cc:597] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 -> Peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Couldn't send request to peer df3a0d1619ee48e8ab68ee0543a72a29. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:36.458074 22991 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.521919 22804 raft_consensus.cc:2393] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 1: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
W20251026 14:58:36.546360 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 70145.9us
I20251026 14:58:36.553509 22806 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:36.556813 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:36.566473 22849 tablet_bootstrap.cc:654] T a673ec034b704e8e86556afe9322d8c2 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:36.566743 22848 tablet_bootstrap.cc:492] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:36.567222 22848 ts_tablet_manager.cc:1403] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.286s user 0.029s sys 0.019s
I20251026 14:58:36.570084 23126 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:36.570207 22991 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:36.576542 22848 raft_consensus.cc:359] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.572753 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:36.579931 22697 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:36.572804 23078 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.584547 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:36.589495 22806 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:36.589632 22697 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:58:36.586289 22774 leader_election.cc:343] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:36.590485 22848 raft_consensus.cc:385] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:36.590623 23132 raft_consensus.cc:2749] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251026 14:58:36.590870 22848 raft_consensus.cc:740] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:36.586969 23145 consensus_queue.cc:1048] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:36.593444 23078 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:36.591095 23032 raft_consensus.cc:2804] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:36.594210 23032 raft_consensus.cc:697] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:36.595991 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:36.596664 22732 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:36.599546 22697 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:36.600687 22991 raft_consensus.cc:2804] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:36.601070 22991 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:36.601436 22991 raft_consensus.cc:3060] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:36.591630 22850 tablet_bootstrap.cc:654] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:36.595057 23032 consensus_queue.cc:237] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:36.586597 22604 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251026 14:58:36.594400 22848 consensus_queue.cc:260] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.612931 22658 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:36.617544 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:58:36.620456 22774 leader_election.cc:343] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:36.619125 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:36.627627 22729 raft_consensus.cc:2393] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 1: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:36.627692 22849 tablet_bootstrap.cc:492] T a673ec034b704e8e86556afe9322d8c2 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:36.628388 22849 ts_tablet_manager.cc:1403] T a673ec034b704e8e86556afe9322d8c2 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.209s user 0.017s sys 0.004s
I20251026 14:58:36.635648 22773 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:36.637187 23145 raft_consensus.cc:2749] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251026 14:58:36.638638 22658 raft_consensus.cc:3060] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:36.640091 22848 ts_tablet_manager.cc:1434] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.073s user 0.004s sys 0.007s
I20251026 14:58:36.641012 22848 tablet_bootstrap.cc:492] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:36.642844 22849 raft_consensus.cc:359] T a673ec034b704e8e86556afe9322d8c2 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.643478 22849 raft_consensus.cc:385] T a673ec034b704e8e86556afe9322d8c2 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:36.643679 22849 raft_consensus.cc:740] T a673ec034b704e8e86556afe9322d8c2 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:36.644168 22849 consensus_queue.cc:260] T a673ec034b704e8e86556afe9322d8c2 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.647812 22732 raft_consensus.cc:2468] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:36.658874 22773 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:36.656302 22991 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.664728 23148 raft_consensus.cc:2804] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:36.665367 23148 raft_consensus.cc:697] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:36.666184 23148 consensus_queue.cc:237] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.666879 22848 tablet_bootstrap.cc:654] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:36.667047 22991 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:36.671979 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:36.672825 22849 ts_tablet_manager.cc:1434] T a673ec034b704e8e86556afe9322d8c2 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.044s user 0.006s sys 0.000s
I20251026 14:58:36.673985 22849 tablet_bootstrap.cc:492] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:36.673058 22806 raft_consensus.cc:3060] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:36.676376 22895 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:36.676858 22895 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.681967 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:36.684372 22895 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:36.685395 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:36.687927 22622 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:36.689635 22895 raft_consensus.cc:2749] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:36.692230 22849 tablet_bootstrap.cc:654] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:36.693507 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:36.697806 22654 raft_consensus.cc:3060] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:36.695819 23078 consensus_queue.cc:1048] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:36.701747 22806 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:36.710028 22697 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:36.710944 22991 raft_consensus.cc:2804] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:36.711426 22991 raft_consensus.cc:697] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:36.712208 22991 consensus_queue.cc:237] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.721873 22555 catalog_manager.cc:5649] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:36.724702 22850 tablet_bootstrap.cc:492] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:36.725230 22850 ts_tablet_manager.cc:1403] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.309s user 0.018s sys 0.030s
I20251026 14:58:36.729197 22654 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:36.736735 22849 tablet_bootstrap.cc:492] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:36.737144 22849 ts_tablet_manager.cc:1403] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.063s user 0.012s sys 0.006s
I20251026 14:58:36.740190 22849 raft_consensus.cc:359] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.740887 22849 raft_consensus.cc:385] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:36.741191 22849 raft_consensus.cc:740] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:36.740681 22552 catalog_manager.cc:5649] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 0 to 1, leader changed from <none> to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:36.741957 22849 consensus_queue.cc:260] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.742588 22850 raft_consensus.cc:359] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.742741 22552 catalog_manager.cc:5649] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 0 to 1, leader changed from <none> to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:36.743304 22850 raft_consensus.cc:385] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:36.743937 22850 raft_consensus.cc:740] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:36.744571 22850 consensus_queue.cc:260] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.747037 22850 ts_tablet_manager.cc:1434] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20251026 14:58:36.747984 22850 tablet_bootstrap.cc:492] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:36.753820 22850 tablet_bootstrap.cc:654] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:36.755100 23145 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:36.755407 22848 tablet_bootstrap.cc:492] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:36.758998 22848 ts_tablet_manager.cc:1403] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.118s user 0.009s sys 0.015s
I20251026 14:58:36.755620 23145 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.761504 22848 raft_consensus.cc:359] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.762236 22848 raft_consensus.cc:385] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:36.762168 23145 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:36.763290 22848 raft_consensus.cc:740] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:36.763407 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:36.766461 22848 consensus_queue.cc:260] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.768795 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:36.770650 22848 ts_tablet_manager.cc:1434] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.011s user 0.005s sys 0.002s
I20251026 14:58:36.771688 22848 tablet_bootstrap.cc:492] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:36.773386 22850 tablet_bootstrap.cc:492] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:36.773921 22850 ts_tablet_manager.cc:1403] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.026s user 0.019s sys 0.001s
I20251026 14:58:36.776582 22850 raft_consensus.cc:359] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.777259 22850 raft_consensus.cc:385] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:36.777510 22850 raft_consensus.cc:740] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:36.777663 22848 tablet_bootstrap.cc:654] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:36.778062 22850 consensus_queue.cc:260] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.779714 22850 ts_tablet_manager.cc:1434] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251026 14:58:36.780459 22850 tablet_bootstrap.cc:492] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:36.781128 23145 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:36.781647 23145 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.784072 23145 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:58:36.785939 22774 leader_election.cc:343] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:36.786202 22850 tablet_bootstrap.cc:654] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
W20251026 14:58:36.792510 22773 leader_election.cc:343] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:36.793011 22773 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:36.793145 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:36.793778 23145 raft_consensus.cc:2749] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:36.794370 22732 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:36.795063 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:36.796166 22773 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:36.797480 23145 raft_consensus.cc:2804] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:36.797820 23145 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:36.798259 23145 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:36.800832 22849 ts_tablet_manager.cc:1434] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.063s user 0.003s sys 0.002s
I20251026 14:58:36.802111 22849 tablet_bootstrap.cc:492] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
W20251026 14:58:36.806070 22774 leader_election.cc:343] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:36.793627 22848 tablet_bootstrap.cc:492] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:36.810223 22848 ts_tablet_manager.cc:1403] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.007s
I20251026 14:58:36.812744 22848 raft_consensus.cc:359] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.813459 22848 raft_consensus.cc:385] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:36.813745 22848 raft_consensus.cc:740] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:36.821096 23145 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.814364 22848 consensus_queue.cc:260] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.825207 22848 ts_tablet_manager.cc:1434] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20251026 14:58:36.826468 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:36.828586 23145 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:36.829586 22848 tablet_bootstrap.cc:492] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:36.829638 22850 tablet_bootstrap.cc:492] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:36.830185 22850 ts_tablet_manager.cc:1403] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.050s user 0.020s sys 0.023s
I20251026 14:58:36.830196 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:36.831089 22732 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:36.833083 22850 raft_consensus.cc:359] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.833810 22850 raft_consensus.cc:385] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:36.834081 22850 raft_consensus.cc:740] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:36.834851 22850 consensus_queue.cc:260] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.836434 22848 tablet_bootstrap.cc:654] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:36.836576 22850 ts_tablet_manager.cc:1434] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251026 14:58:36.838706 22732 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:36.839985 22773 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:36.840823 23151 raft_consensus.cc:2804] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:36.841297 23151 raft_consensus.cc:697] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:36.842082 23151 consensus_queue.cc:237] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.842597 22850 tablet_bootstrap.cc:492] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:36.849552 22850 tablet_bootstrap.cc:654] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:36.857905 22654 raft_consensus.cc:3060] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
W20251026 14:58:36.866756 22774 leader_election.cc:343] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:36.879576 22555 catalog_manager.cc:5649] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:36.901042 23133 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:36.901645 23133 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.903636 23133 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:36.904708 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:36.909523 22732 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
W20251026 14:58:36.910481 22700 leader_election.cc:343] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:36.915462 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:58:36.916585 22697 leader_election.cc:343] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:36.917024 22697 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:36.917769 23132 raft_consensus.cc:2749] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:36.922137 22850 tablet_bootstrap.cc:492] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:36.922612 22850 ts_tablet_manager.cc:1403] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.080s user 0.026s sys 0.034s
I20251026 14:58:36.925220 23132 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:36.925763 23132 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.928792 23132 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:36.929900 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:36.930696 22849 tablet_bootstrap.cc:654] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:36.930521 22850 raft_consensus.cc:359] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.930938 22848 tablet_bootstrap.cc:492] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:36.931232 22850 raft_consensus.cc:385] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:36.931481 22848 ts_tablet_manager.cc:1403] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.106s user 0.030s sys 0.034s
W20251026 14:58:36.931519 22700 leader_election.cc:343] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:36.932420 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:36.933084 22658 raft_consensus.cc:3060] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:36.933038 22804 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:36.934393 22848 raft_consensus.cc:359] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.935043 22848 raft_consensus.cc:385] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:36.935297 22848 raft_consensus.cc:740] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:36.936515 22848 consensus_queue.cc:260] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.937472 22697 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:36.938858 22991 raft_consensus.cc:2804] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:36.939378 22991 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:36.939707 22848 ts_tablet_manager.cc:1434] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20251026 14:58:36.939922 22991 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:36.940950 22848 tablet_bootstrap.cc:492] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:36.931614 22850 raft_consensus.cc:740] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:36.943353 22850 consensus_queue.cc:260] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.945991 22850 ts_tablet_manager.cc:1434] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.023s user 0.006s sys 0.001s
I20251026 14:58:36.947280 22850 tablet_bootstrap.cc:492] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:36.948393 22848 tablet_bootstrap.cc:654] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:36.953581 22991 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.963280 22991 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:36.964368 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:58:36.966403 22700 leader_election.cc:343] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:36.968716 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:36.969620 22804 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:36.970256 22850 tablet_bootstrap.cc:654] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:36.971776 22849 tablet_bootstrap.cc:492] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:36.972347 22849 ts_tablet_manager.cc:1403] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.170s user 0.006s sys 0.006s
I20251026 14:58:36.986340 22804 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:36.987391 22849 raft_consensus.cc:359] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.987898 22849 raft_consensus.cc:385] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:36.988094 22849 raft_consensus.cc:740] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:36.988564 22849 consensus_queue.cc:260] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.989768 22697 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:36.990515 22991 raft_consensus.cc:2804] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:36.990581 22849 ts_tablet_manager.cc:1434] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.018s user 0.004s sys 0.000s
I20251026 14:58:36.991147 22991 raft_consensus.cc:697] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:36.992530 22991 consensus_queue.cc:237] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:36.996968 22849 tablet_bootstrap.cc:492] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:37.002424 23132 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:37.002990 23132 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.005012 23132 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:37.005934 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:37.006506 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:58:37.007344 22700 leader_election.cc:343] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:37.009044 22850 tablet_bootstrap.cc:492] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:37.009563 22850 ts_tablet_manager.cc:1403] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.063s user 0.011s sys 0.006s
I20251026 14:58:37.009791 22554 catalog_manager.cc:5649] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 0 to 1, leader changed from <none> to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:37.011657 22848 tablet_bootstrap.cc:492] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:37.012188 22848 ts_tablet_manager.cc:1403] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.071s user 0.020s sys 0.008s
W20251026 14:58:37.013005 22697 leader_election.cc:343] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:37.013484 22697 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:37.014279 23132 raft_consensus.cc:2749] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:37.015030 22848 raft_consensus.cc:359] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.015777 22848 raft_consensus.cc:385] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:37.012188 22850 raft_consensus.cc:359] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.016036 22848 raft_consensus.cc:740] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:37.016381 22850 raft_consensus.cc:385] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:37.016678 22850 raft_consensus.cc:740] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:37.016942 22848 consensus_queue.cc:260] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.017433 22850 consensus_queue.cc:260] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.019726 22850 ts_tablet_manager.cc:1434] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20251026 14:58:37.020082 22848 ts_tablet_manager.cc:1434] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20251026 14:58:37.021129 22848 tablet_bootstrap.cc:492] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:37.027041 22850 tablet_bootstrap.cc:492] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:37.035362 22848 tablet_bootstrap.cc:654] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:37.037108 22849 tablet_bootstrap.cc:654] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:37.043067 22850 tablet_bootstrap.cc:654] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:37.050678 22850 tablet_bootstrap.cc:492] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:37.051214 22850 ts_tablet_manager.cc:1403] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.000s
I20251026 14:58:37.072942 23132 consensus_queue.cc:1048] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:37.077689 22849 tablet_bootstrap.cc:492] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:37.078186 22849 ts_tablet_manager.cc:1403] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.081s user 0.031s sys 0.008s
I20251026 14:58:37.082419 22848 tablet_bootstrap.cc:492] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:37.082996 22848 ts_tablet_manager.cc:1403] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.062s user 0.025s sys 0.020s
I20251026 14:58:37.083447 22849 raft_consensus.cc:359] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.084210 22849 raft_consensus.cc:385] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:37.083876 22850 raft_consensus.cc:359] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.084486 22849 raft_consensus.cc:740] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:37.084575 22850 raft_consensus.cc:385] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:37.084919 22850 raft_consensus.cc:740] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:37.085181 22849 consensus_queue.cc:260] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.085655 22850 consensus_queue.cc:260] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.087420 22849 ts_tablet_manager.cc:1434] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20251026 14:58:37.088676 22849 tablet_bootstrap.cc:492] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:37.094735 22848 raft_consensus.cc:359] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.095525 22848 raft_consensus.cc:385] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:37.095757 22848 raft_consensus.cc:740] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:37.096315 22848 consensus_queue.cc:260] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.098852 22850 ts_tablet_manager.cc:1434] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.047s user 0.009s sys 0.000s
I20251026 14:58:37.100535 22848 ts_tablet_manager.cc:1434] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.017s user 0.002s sys 0.004s
I20251026 14:58:37.101714 22848 tablet_bootstrap.cc:492] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:37.102206 22849 tablet_bootstrap.cc:654] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:37.108175 22848 tablet_bootstrap.cc:654] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:37.108572 23122 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:37.109117 23122 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.110258 22850 tablet_bootstrap.cc:492] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:37.111119 23122 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:37.114288 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:37.115129 22804 raft_consensus.cc:2393] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 1: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:37.119132 22850 tablet_bootstrap.cc:654] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:37.127060 22849 tablet_bootstrap.cc:492] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:37.127666 22849 ts_tablet_manager.cc:1403] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.039s user 0.015s sys 0.013s
I20251026 14:58:37.130841 22849 raft_consensus.cc:359] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.131980 22849 raft_consensus.cc:385] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:37.132505 22849 raft_consensus.cc:740] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:37.133422 22849 consensus_queue.cc:260] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.134217 22991 consensus_queue.cc:1048] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:37.135777 22849 ts_tablet_manager.cc:1434] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20251026 14:58:37.136946 22849 tablet_bootstrap.cc:492] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:37.140013 22850 tablet_bootstrap.cc:492] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:37.140549 22850 ts_tablet_manager.cc:1403] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.008s
W20251026 14:58:37.151305 22700 consensus_peers.cc:597] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e -> Peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Couldn't send request to peer df3a0d1619ee48e8ab68ee0543a72a29. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:37.155128 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:37.156023 22728 raft_consensus.cc:2393] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 1: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:37.161821 22624 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:37.165833 23122 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:37.174469 22849 tablet_bootstrap.cc:654] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:37.175855 22850 raft_consensus.cc:359] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.189865 22850 raft_consensus.cc:385] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:37.190212 22850 raft_consensus.cc:740] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:37.190829 22850 consensus_queue.cc:260] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.202527 22850 ts_tablet_manager.cc:1434] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.062s user 0.002s sys 0.006s
I20251026 14:58:37.204450 22658 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:37.210918 22850 tablet_bootstrap.cc:492] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:37.204629 22848 tablet_bootstrap.cc:492] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:37.214100 22848 ts_tablet_manager.cc:1403] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.113s user 0.022s sys 0.039s
I20251026 14:58:37.221405 23122 raft_consensus.cc:2749] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:37.226835 22991 consensus_queue.cc:1048] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:37.221833 23078 consensus_queue.cc:1048] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20251026 14:58:37.235587 23078 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:37.238680 23078 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.244671 23078 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:37.244622 22848 raft_consensus.cc:359] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.246927 22848 raft_consensus.cc:385] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:37.247225 22848 raft_consensus.cc:740] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:37.245239 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:37.247818 22848 consensus_queue.cc:260] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.251381 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:37.252022 22732 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:37.255913 22850 tablet_bootstrap.cc:654] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:37.259696 22849 tablet_bootstrap.cc:492] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:37.259651 22659 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:37.260401 22849 ts_tablet_manager.cc:1403] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.124s user 0.014s sys 0.013s
I20251026 14:58:37.262663 22849 raft_consensus.cc:359] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.263144 22849 raft_consensus.cc:385] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:37.263402 22849 raft_consensus.cc:740] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:37.263878 22849 consensus_queue.cc:260] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.265331 22848 ts_tablet_manager.cc:1434] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.051s user 0.007s sys 0.000s
I20251026 14:58:37.267345 22848 tablet_bootstrap.cc:492] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:37.272166 22991 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:37.272735 22991 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.273527 22850 tablet_bootstrap.cc:492] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:37.274664 22991 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:37.275184 22850 ts_tablet_manager.cc:1403] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.064s user 0.009s sys 0.002s
I20251026 14:58:37.282925 22850 raft_consensus.cc:359] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.283511 22849 ts_tablet_manager.cc:1434] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.023s user 0.005s sys 0.001s
I20251026 14:58:37.284549 22848 tablet_bootstrap.cc:654] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:37.284842 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:37.283607 22850 raft_consensus.cc:385] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:37.277931 22774 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:37.285233 22849 tablet_bootstrap.cc:492] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:37.286633 22850 raft_consensus.cc:740] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:37.294567 22850 consensus_queue.cc:260] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.296978 22850 ts_tablet_manager.cc:1434] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20251026 14:58:37.288142 23148 raft_consensus.cc:2804] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:37.297698 23148 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:37.297968 22850 tablet_bootstrap.cc:492] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:37.298058 23148 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:37.301559 22849 tablet_bootstrap.cc:654] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:37.290571 23145 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:37.302381 23145 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.304389 23145 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:37.286763 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:37.308095 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:37.309000 22700 leader_election.cc:343] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
W20251026 14:58:37.309290 22774 leader_election.cc:343] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:37.285503 22805 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:37.309194 22850 tablet_bootstrap.cc:654] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:37.311020 22697 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:37.311933 23133 raft_consensus.cc:2804] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:37.312247 23133 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:37.312556 23133 raft_consensus.cc:3060] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:37.318289 22850 tablet_bootstrap.cc:492] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:37.318806 22850 ts_tablet_manager.cc:1403] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.004s
I20251026 14:58:37.328392 23133 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.328938 23148 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:37.325317 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 47237.7us
I20251026 14:58:37.330628 23133 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:37.337507 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:37.337996 22806 raft_consensus.cc:3060] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:37.339630 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:37.335041 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:37.342394 22729 raft_consensus.cc:2468] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:37.344357 22991 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:37.344657 23148 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:37.344830 22991 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.342453 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:37.342034 23145 consensus_queue.cc:1048] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20251026 14:58:37.343552 22700 leader_election.cc:343] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:37.346743 22991 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:37.358728 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:37.360231 22700 leader_election.cc:343] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:37.360687 22655 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:37.363494 22806 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:37.360597 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:37.364496 22804 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:37.364559 22697 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:37.365653 23032 raft_consensus.cc:2804] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:37.366129 23032 raft_consensus.cc:697] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:37.366976 23032 consensus_queue.cc:237] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.367257 22850 raft_consensus.cc:359] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.367949 22850 raft_consensus.cc:385] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:37.368225 22850 raft_consensus.cc:740] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:37.369222 22849 tablet_bootstrap.cc:492] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:37.373842 22849 ts_tablet_manager.cc:1403] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.089s user 0.013s sys 0.004s
I20251026 14:58:37.379505 22848 tablet_bootstrap.cc:492] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:37.380339 22848 ts_tablet_manager.cc:1403] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.113s user 0.031s sys 0.028s
I20251026 14:58:37.368860 22850 consensus_queue.cc:260] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.370478 22697 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:37.380873 22895 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:37.381426 22895 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.382989 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:37.383378 22895 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:37.383603 22729 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:37.383770 22655 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:37.386240 22774 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:37.386714 22850 ts_tablet_manager.cc:1434] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.068s user 0.007s sys 0.000s
I20251026 14:58:37.387660 22850 tablet_bootstrap.cc:492] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:37.387730 22848 raft_consensus.cc:359] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.390401 22848 raft_consensus.cc:385] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:37.390738 22848 raft_consensus.cc:740] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:37.391389 22773 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:37.391517 22848 consensus_queue.cc:260] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.392262 23148 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:37.392727 23148 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.393254 23151 consensus_queue.cc:1048] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:37.394541 22848 ts_tablet_manager.cc:1434] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20251026 14:58:37.394608 23148 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:37.394697 23151 raft_consensus.cc:2804] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:37.395244 23151 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:37.395530 22991 raft_consensus.cc:2804] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:37.395627 23151 raft_consensus.cc:3060] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:37.395978 22991 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:37.396368 22991 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:37.392287 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:37.398248 22805 raft_consensus.cc:2393] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 1: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:37.399055 23148 raft_consensus.cc:2804] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:37.399617 23148 raft_consensus.cc:697] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:37.400379 22850 tablet_bootstrap.cc:654] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:37.400512 23148 consensus_queue.cc:237] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.402813 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:37.402812 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:37.404528 22734 raft_consensus.cc:2393] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 1: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:37.405874 22848 tablet_bootstrap.cc:492] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
W20251026 14:58:37.407779 22774 leader_election.cc:343] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:37.415840 22624 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:37.417500 22895 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
W20251026 14:58:37.419373 22774 consensus_peers.cc:597] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 -> Peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Couldn't send request to peer df3a0d1619ee48e8ab68ee0543a72a29. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20251026 14:58:37.421823 22700 consensus_peers.cc:597] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e -> Peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Couldn't send request to peer df3a0d1619ee48e8ab68ee0543a72a29. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:37.423656 22849 raft_consensus.cc:359] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.425422 22849 raft_consensus.cc:385] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:37.426098 22849 raft_consensus.cc:740] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:37.429183 22895 raft_consensus.cc:2749] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:37.431286 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:37.430778 22849 consensus_queue.cc:260] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.434867 22734 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:37.437981 22773 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:37.439371 22848 tablet_bootstrap.cc:654] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:37.441099 22895 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:37.441604 22895 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.446405 22849 ts_tablet_manager.cc:1434] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.072s user 0.005s sys 0.002s
I20251026 14:58:37.446666 22895 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:37.447286 22849 tablet_bootstrap.cc:492] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:37.442479 23189 raft_consensus.cc:2804] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:37.474123 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:37.468456 23189 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:37.481622 23189 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:37.461953 22729 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:37.515084 23133 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:37.495338 23151 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.606510 22554 catalog_manager.cc:5649] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:37.544226 23133 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.579754 23080 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:37.561655 22991 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.670655 23133 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:37.631645 23080 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.684286 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:37.685359 22806 raft_consensus.cc:2393] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 1: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:37.692216 23080 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:37.709595 22991 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:37.705137 23151 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:37.731446 22848 tablet_bootstrap.cc:492] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:37.731981 22848 ts_tablet_manager.cc:1403] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.330s user 0.019s sys 0.014s
I20251026 14:58:37.734979 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:37.718068 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:37.741075 22848 raft_consensus.cc:359] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.748921 22848 raft_consensus.cc:385] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:37.749429 22848 raft_consensus.cc:740] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:37.750631 22848 consensus_queue.cc:260] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.755591 22850 tablet_bootstrap.cc:492] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:37.756299 22850 ts_tablet_manager.cc:1403] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.369s user 0.021s sys 0.010s
W20251026 14:58:37.766950 22604 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251026 14:58:37.739127 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:37.768855 22848 ts_tablet_manager.cc:1434] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.037s user 0.008s sys 0.000s
I20251026 14:58:37.769999 22848 tablet_bootstrap.cc:492] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:37.776706 22849 tablet_bootstrap.cc:654] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
W20251026 14:58:37.778650 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 77575.5us
I20251026 14:58:37.793663 22991 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:37.815096 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:37.809900 22991 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:37.816584 23192 log.cc:927] Time spent T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.270s user 0.004s sys 0.000s
I20251026 14:58:37.837922 22848 tablet_bootstrap.cc:654] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:37.838575 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:37.947355 22622 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:58:37.951284 22700 leader_election.cc:343] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
W20251026 14:58:37.952311 22700 leader_election.cc:343] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
W20251026 14:58:37.951464 22774 leader_election.cc:343] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:37.905777 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:37.953913 23122 raft_consensus.cc:3060] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:37.956703 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:37.969085 22850 raft_consensus.cc:359] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.969872 22850 raft_consensus.cc:385] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:37.970142 22850 raft_consensus.cc:740] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
W20251026 14:58:37.959476 23191 log.cc:927] Time spent T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.225s user 0.001s sys 0.001s
I20251026 14:58:37.964713 22805 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:37.965330 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:37.961103 23189 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.971711 22555 catalog_manager.cc:5649] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 0 to 1, leader changed from <none> to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:37.970746 22850 consensus_queue.cc:260] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:37.975255 22803 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:37.995793 22850 ts_tablet_manager.cc:1434] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.239s user 0.010s sys 0.009s
I20251026 14:58:37.996555 22850 tablet_bootstrap.cc:492] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
W20251026 14:58:37.997984 22697 leader_election.cc:343] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:37.998548 22697 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:37.999934 22697 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:37.984863 22730 raft_consensus.cc:3060] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:38.001979 23211 raft_consensus.cc:2804] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:38.003976 23211 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:38.004330 23211 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:37.992233 22991 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:38.020963 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:38.023377 22849 tablet_bootstrap.cc:492] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:38.002537 23196 raft_consensus.cc:2749] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:38.023908 22849 ts_tablet_manager.cc:1403] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.577s user 0.012s sys 0.006s
I20251026 14:58:37.991840 23032 consensus_queue.cc:1048] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
I20251026 14:58:38.024453 22991 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:38.035550 23189 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:38.055521 23122 raft_consensus.cc:2749] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:38.074446 22654 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:38.048748 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:58:38.042799 22697 leader_election.cc:343] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:38.075595 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:38.083467 22727 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:38.085911 23198 consensus_queue.cc:1048] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251026 14:58:38.102394 22849 raft_consensus.cc:359] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.103513 22849 raft_consensus.cc:385] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:38.104658 22849 raft_consensus.cc:740] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:38.105602 22849 consensus_queue.cc:260] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:38.109440 22700 leader_election.cc:343] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:38.042805 22991 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.042193 23196 consensus_queue.cc:1048] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.053s
I20251026 14:58:38.137177 22849 ts_tablet_manager.cc:1434] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.111s user 0.007s sys 0.000s
I20251026 14:58:38.138850 22849 tablet_bootstrap.cc:492] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:38.141744 23189 consensus_queue.cc:1048] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 14:58:38.148077 22774 leader_election.cc:343] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:38.178009 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:38.184607 22803 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:38.187058 22697 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:38.190511 23196 raft_consensus.cc:2804] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:38.195952 23196 raft_consensus.cc:697] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:38.196867 23196 consensus_queue.cc:237] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.217397 22991 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:38.221556 22850 tablet_bootstrap.cc:654] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:38.232867 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:38.234025 22805 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:38.235468 22697 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:38.221015 23211 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.231837 22727 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:38.239565 22773 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:38.241557 23213 raft_consensus.cc:2804] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:38.242655 23213 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:38.242964 23213 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:38.243466 23218 raft_consensus.cc:2804] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:38.245679 23218 raft_consensus.cc:697] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:38.249187 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:38.249764 22730 raft_consensus.cc:2468] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:38.252530 22773 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:38.246529 23218 consensus_queue.cc:237] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.253456 22848 tablet_bootstrap.cc:492] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:38.253926 22848 ts_tablet_manager.cc:1403] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.484s user 0.025s sys 0.018s
I20251026 14:58:38.250401 22652 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
W20251026 14:58:38.256723 22700 leader_election.cc:343] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:38.259140 23211 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:38.257210 22700 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:38.265573 23189 raft_consensus.cc:2804] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:38.266088 23211 raft_consensus.cc:2749] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:38.279384 23201 consensus_queue.cc:1048] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20251026 14:58:38.273530 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:38.286365 22850 tablet_bootstrap.cc:492] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:38.286890 22850 ts_tablet_manager.cc:1403] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.290s user 0.008s sys 0.005s
I20251026 14:58:38.278398 23189 raft_consensus.cc:697] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:38.290777 22848 raft_consensus.cc:359] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.291464 22848 raft_consensus.cc:385] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:38.291697 22848 raft_consensus.cc:740] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:38.293051 22849 tablet_bootstrap.cc:654] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:38.292308 22848 consensus_queue.cc:260] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.287501 22804 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:38.303288 22850 raft_consensus.cc:359] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.303833 22850 raft_consensus.cc:385] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:38.304087 22850 raft_consensus.cc:740] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:38.310678 23189 consensus_queue.cc:237] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.317101 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:38.318986 22848 ts_tablet_manager.cc:1434] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.065s user 0.006s sys 0.002s
I20251026 14:58:38.319968 22848 tablet_bootstrap.cc:492] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:38.325045 23213 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.329821 22850 consensus_queue.cc:260] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.335408 22804 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:38.338163 22697 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:38.347915 22850 ts_tablet_manager.cc:1434] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.061s user 0.009s sys 0.002s
I20251026 14:58:38.349320 22850 tablet_bootstrap.cc:492] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:38.351737 22849 tablet_bootstrap.cc:492] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:38.352473 22849 ts_tablet_manager.cc:1403] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.214s user 0.010s sys 0.011s
W20251026 14:58:38.352607 22700 leader_election.cc:343] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:38.357208 22658 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:38.347260 23201 raft_consensus.cc:2804] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:38.358621 23201 raft_consensus.cc:697] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:38.359508 23201 consensus_queue.cc:237] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.361358 22849 raft_consensus.cc:359] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.366919 22849 raft_consensus.cc:385] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:38.367280 22849 raft_consensus.cc:740] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:38.368140 22849 consensus_queue.cc:260] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.364607 23213 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:38.373337 22849 ts_tablet_manager.cc:1434] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.020s user 0.010s sys 0.002s
I20251026 14:58:38.376394 22849 tablet_bootstrap.cc:492] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:38.378695 22848 tablet_bootstrap.cc:654] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:38.392855 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:38.385311 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:38.402200 23213 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:38.403332 23213 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.401850 23204 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:38.401237 22804 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:38.406605 23204 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.408830 23213 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:38.408203 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:38.397475 22652 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
W20251026 14:58:38.414448 22697 leader_election.cc:343] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:38.427148 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:38.443303 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:38.444540 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:38.447660 22731 raft_consensus.cc:2468] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:38.420897 23204 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:38.450909 22850 tablet_bootstrap.cc:654] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
W20251026 14:58:38.448734 22700 leader_election.cc:343] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:38.451752 22700 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:38.452051 22849 tablet_bootstrap.cc:654] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:38.453562 23201 raft_consensus.cc:2749] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 14:58:38.454859 22774 leader_election.cc:343] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:38.456941 22773 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:38.459192 23198 raft_consensus.cc:2804] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:38.459645 23198 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:38.459991 23198 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:38.470705 22804 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:38.475718 22697 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:38.477530 23196 raft_consensus.cc:2804] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:38.478060 23196 raft_consensus.cc:697] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 2 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:38.472055 23198 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.478893 23196 consensus_queue.cc:237] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.485095 23198 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:38.490806 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:38.491479 22730 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:38.494874 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:58:38.501338 22774 leader_election.cc:343] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:38.540663 22730 raft_consensus.cc:2468] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:38.558506 22773 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:38.562671 22652 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:38.563824 23198 raft_consensus.cc:2804] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:38.564471 23198 raft_consensus.cc:697] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:38.565302 23198 consensus_queue.cc:237] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.570148 22848 tablet_bootstrap.cc:492] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:38.570684 22848 ts_tablet_manager.cc:1403] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.251s user 0.017s sys 0.010s
I20251026 14:58:38.574369 22848 raft_consensus.cc:359] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.575937 22848 raft_consensus.cc:385] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:38.577252 22848 raft_consensus.cc:740] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:38.578747 22848 consensus_queue.cc:260] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.581156 22849 tablet_bootstrap.cc:492] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:38.581713 22849 ts_tablet_manager.cc:1403] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.206s user 0.027s sys 0.005s
I20251026 14:58:38.582670 22848 ts_tablet_manager.cc:1434] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.012s user 0.002s sys 0.004s
I20251026 14:58:38.583725 22848 tablet_bootstrap.cc:492] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:38.584255 22849 raft_consensus.cc:359] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.584640 23204 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:38.585143 23204 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.570071 22850 tablet_bootstrap.cc:492] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:38.587004 22850 ts_tablet_manager.cc:1403] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.238s user 0.019s sys 0.026s
I20251026 14:58:38.588243 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:38.588922 22730 raft_consensus.cc:2468] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:38.584957 22849 raft_consensus.cc:385] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:38.590032 22849 raft_consensus.cc:740] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:38.590193 22773 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:38.590672 22849 consensus_queue.cc:260] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.591087 23197 raft_consensus.cc:2804] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:38.591480 23197 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:38.590868 22554 catalog_manager.cc:5649] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:38.591806 23197 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:38.592931 22849 ts_tablet_manager.cc:1434] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20251026 14:58:38.592571 22554 catalog_manager.cc:5649] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:38.593873 22849 tablet_bootstrap.cc:492] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:38.599810 23204 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:38.600714 22849 tablet_bootstrap.cc:654] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:38.601657 22850 raft_consensus.cc:359] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.602454 22850 raft_consensus.cc:385] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:38.602735 22850 raft_consensus.cc:740] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:38.607447 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:38.607736 23197 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.608201 22652 raft_consensus.cc:2468] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:38.609635 22850 consensus_queue.cc:260] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.612475 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:38.612603 22850 ts_tablet_manager.cc:1434] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.025s user 0.010s sys 0.001s
I20251026 14:58:38.613077 22730 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:38.614042 22850 tablet_bootstrap.cc:492] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:38.614579 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:38.615187 22652 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:38.622696 22850 tablet_bootstrap.cc:654] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:38.609700 22848 tablet_bootstrap.cc:654] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:38.625697 22730 raft_consensus.cc:2468] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:38.626055 22652 raft_consensus.cc:2468] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:38.624439 23197 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:38.627163 22773 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:38.628022 23197 raft_consensus.cc:2804] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:38.628448 23197 raft_consensus.cc:697] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:38.629190 23197 consensus_queue.cc:237] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.642524 22555 catalog_manager.cc:5649] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 0 to 2, leader changed from <none> to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 2 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:38.645529 22848 tablet_bootstrap.cc:492] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:38.646265 22848 ts_tablet_manager.cc:1403] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.063s user 0.013s sys 0.000s
I20251026 14:58:38.645736 22555 catalog_manager.cc:5649] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 0 to 1, leader changed from <none> to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:38.647293 22555 catalog_manager.cc:5649] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 0 to 1, leader changed from <none> to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:38.650689 22849 tablet_bootstrap.cc:492] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:38.651175 22849 ts_tablet_manager.cc:1403] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.057s user 0.031s sys 0.012s
I20251026 14:58:38.655478 22850 tablet_bootstrap.cc:492] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:38.656049 22850 ts_tablet_manager.cc:1403] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.042s user 0.013s sys 0.000s
I20251026 14:58:38.648747 22848 raft_consensus.cc:359] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.657419 22848 raft_consensus.cc:385] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:38.657757 22848 raft_consensus.cc:740] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:38.658448 22848 consensus_queue.cc:260] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.660148 22848 ts_tablet_manager.cc:1434] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20251026 14:58:38.662137 22850 raft_consensus.cc:359] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.662868 22850 raft_consensus.cc:385] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:38.663136 22850 raft_consensus.cc:740] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:38.663754 22850 consensus_queue.cc:260] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.666025 22848 tablet_bootstrap.cc:492] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:38.673803 22849 raft_consensus.cc:359] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.674741 22849 raft_consensus.cc:385] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:38.675211 22849 raft_consensus.cc:740] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:38.675889 22849 consensus_queue.cc:260] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.667826 22850 ts_tablet_manager.cc:1434] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20251026 14:58:38.678315 22849 ts_tablet_manager.cc:1434] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.027s user 0.005s sys 0.003s
I20251026 14:58:38.678511 22850 tablet_bootstrap.cc:492] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:38.679276 22849 tablet_bootstrap.cc:492] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:38.678329 22848 tablet_bootstrap.cc:654] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:38.685456 22850 tablet_bootstrap.cc:654] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:38.690364 22849 tablet_bootstrap.cc:654] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:38.702358 22849 tablet_bootstrap.cc:492] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:38.702862 22849 ts_tablet_manager.cc:1403] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.012s
I20251026 14:58:38.705633 22849 raft_consensus.cc:359] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.706652 22849 raft_consensus.cc:385] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:38.706939 22849 raft_consensus.cc:740] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:38.708117 22849 consensus_queue.cc:260] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.710037 22849 ts_tablet_manager.cc:1434] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20251026 14:58:38.712898 22849 tablet_bootstrap.cc:492] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:38.714628 23196 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:38.715165 23196 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.717175 23196 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:38.717489 22848 tablet_bootstrap.cc:492] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:38.718164 22848 ts_tablet_manager.cc:1403] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.052s user 0.010s sys 0.003s
I20251026 14:58:38.719257 22849 tablet_bootstrap.cc:654] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:38.719498 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:38.721079 22700 leader_election.cc:343] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:38.720801 22554 catalog_manager.cc:5649] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 2, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 2 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:38.721657 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:38.722353 22803 raft_consensus.cc:2468] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:38.722497 22554 catalog_manager.cc:5649] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:38.723953 22697 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:38.724643 23211 raft_consensus.cc:2804] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:38.725021 23211 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:38.724858 22848 raft_consensus.cc:359] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.725464 23211 raft_consensus.cc:3060] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:38.725716 22848 raft_consensus.cc:385] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:38.726019 22848 raft_consensus.cc:740] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:38.726193 23122 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:38.726677 23122 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.726787 22848 consensus_queue.cc:260] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.728580 23122 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:38.728981 22848 ts_tablet_manager.cc:1434] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20251026 14:58:38.729928 22848 tablet_bootstrap.cc:492] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:38.730018 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:38.733824 23211 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.734328 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:38.735054 22850 tablet_bootstrap.cc:492] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:38.735569 22850 ts_tablet_manager.cc:1403] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.057s user 0.012s sys 0.024s
I20251026 14:58:38.735988 22848 tablet_bootstrap.cc:654] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:38.737028 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:38.737380 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:38.737684 22803 raft_consensus.cc:3060] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:38.735996 23211 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:38.738883 22624 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:38.741874 23122 raft_consensus.cc:2749] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:38.743166 22850 raft_consensus.cc:359] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:38.743433 22700 leader_election.cc:343] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:38.744277 22850 raft_consensus.cc:385] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:38.744607 22850 raft_consensus.cc:740] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:38.751627 22803 raft_consensus.cc:2468] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:38.745345 22850 consensus_queue.cc:260] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.753106 22697 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:38.754755 23211 raft_consensus.cc:2804] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:38.755334 23211 raft_consensus.cc:697] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:38.755512 22850 ts_tablet_manager.cc:1434] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20251026 14:58:38.756326 23211 consensus_queue.cc:237] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.762049 22849 tablet_bootstrap.cc:492] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:38.762578 22849 ts_tablet_manager.cc:1403] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.005s
I20251026 14:58:38.778198 22850 tablet_bootstrap.cc:492] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e: Bootstrap starting.
I20251026 14:58:38.797436 23198 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:38.798003 23198 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.799017 22850 tablet_bootstrap.cc:654] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:38.798748 22849 raft_consensus.cc:359] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.799435 22849 raft_consensus.cc:385] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:38.799717 22849 raft_consensus.cc:740] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:38.799993 23198 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:38.802606 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:38.804121 22774 leader_election.cc:343] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:38.800406 22849 consensus_queue.cc:260] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.806017 22849 ts_tablet_manager.cc:1434] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.043s user 0.007s sys 0.000s
I20251026 14:58:38.806946 22849 tablet_bootstrap.cc:492] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:38.812104 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:38.812826 22733 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:38.817916 22773 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:38.826287 22653 raft_consensus.cc:3060] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:38.830657 23196 consensus_queue.cc:1048] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:38.841146 22554 catalog_manager.cc:5649] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 0 to 1, leader changed from <none> to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:38.845119 22850 tablet_bootstrap.cc:492] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e: No bootstrap required, opened a new log
I20251026 14:58:38.845682 22850 ts_tablet_manager.cc:1403] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e: Time spent bootstrapping tablet: real 0.088s user 0.014s sys 0.000s
W20251026 14:58:38.847911 22700 consensus_peers.cc:597] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e -> Peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Couldn't send request to peer df3a0d1619ee48e8ab68ee0543a72a29. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:38.852205 22652 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:38.848475 22850 raft_consensus.cc:359] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.853678 22850 raft_consensus.cc:385] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:38.853912 22850 raft_consensus.cc:740] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Initialized, Role: FOLLOWER
I20251026 14:58:38.853858 23211 consensus_queue.cc:1048] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:38.854588 22850 consensus_queue.cc:260] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.857178 23189 consensus_queue.cc:1048] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:38.856925 23204 consensus_queue.cc:1048] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:38.857002 23197 raft_consensus.cc:2804] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:38.859079 23197 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:38.859423 23197 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:38.863682 23198 raft_consensus.cc:493] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:38.864223 23198 raft_consensus.cc:515] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.865402 22848 tablet_bootstrap.cc:492] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:38.866164 22848 ts_tablet_manager.cc:1403] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.136s user 0.004s sys 0.018s
I20251026 14:58:38.867228 23189 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:38.876838 23189 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.884061 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:38.886014 22774 leader_election.cc:343] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
W20251026 14:58:38.886893 22604 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251026 14:58:38.884999 23198 leader_election.cc:290] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:38.887029 23204 consensus_queue.cc:1048] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:38.893507 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:38.894421 22732 raft_consensus.cc:2468] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:38.891456 22850 ts_tablet_manager.cc:1434] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e: Time spent starting tablet: real 0.046s user 0.005s sys 0.000s
I20251026 14:58:38.895637 22848 raft_consensus.cc:359] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.897217 22848 raft_consensus.cc:385] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:38.897540 22848 raft_consensus.cc:740] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:38.898164 22848 consensus_queue.cc:260] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.900811 22848 ts_tablet_manager.cc:1434] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.034s user 0.006s sys 0.001s
I20251026 14:58:38.903445 22773 leader_election.cc:304] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:38.907074 22848 tablet_bootstrap.cc:492] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:38.911676 23189 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:38.914326 22654 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:38.922716 23148 raft_consensus.cc:2804] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:38.923192 23148 raft_consensus.cc:493] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:38.923542 23148 raft_consensus.cc:3060] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:38.930974 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:38.931324 23203 consensus_queue.cc:1048] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20251026 14:58:38.939958 22774 leader_election.cc:343] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:38.943295 22849 tablet_bootstrap.cc:654] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:38.947069 23148 raft_consensus.cc:515] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:38.948876 22848 tablet_bootstrap.cc:654] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
W20251026 14:58:38.950014 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 46703.6us
I20251026 14:58:39.027276 22849 tablet_bootstrap.cc:492] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:39.030795 22849 ts_tablet_manager.cc:1403] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.224s user 0.021s sys 0.000s
I20251026 14:58:39.032359 23202 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:58:39.029459 22700 consensus_peers.cc:597] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e -> Peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Couldn't send request to peer df3a0d1619ee48e8ab68ee0543a72a29. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20251026 14:58:39.033039 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 82058.1us
I20251026 14:58:39.032856 23202 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:39.033918 22895 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:39.034386 22895 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:39.034902 23202 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:39.036123 22895 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:58:39.036976 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 69480.7us
I20251026 14:58:39.041126 23148 leader_election.cc:290] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:39.045591 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:39.045822 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:39.046169 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:39.046324 22729 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:39.034962 22849 raft_consensus.cc:359] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:39.051190 22849 raft_consensus.cc:385] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:39.051611 22849 raft_consensus.cc:740] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:39.052501 22849 consensus_queue.cc:260] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:39.060557 22774 leader_election.cc:343] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
W20251026 14:58:39.068171 22700 leader_election.cc:343] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:39.082710 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:39.083649 22733 raft_consensus.cc:3060] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:39.095654 22849 ts_tablet_manager.cc:1434] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.064s user 0.006s sys 0.000s
I20251026 14:58:39.099854 22849 tablet_bootstrap.cc:492] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821: Bootstrap starting.
I20251026 14:58:39.057236 23197 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:39.101497 22773 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:39.115120 22848 tablet_bootstrap.cc:492] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:39.115803 22848 ts_tablet_manager.cc:1403] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.209s user 0.017s sys 0.005s
I20251026 14:58:39.098155 23202 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:39.128293 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:39.102490 23032 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:39.134145 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:39.113225 23203 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251026 14:58:39.140560 23259 log.cc:927] Time spent T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.098s user 0.001s sys 0.000s
I20251026 14:58:39.162881 22895 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:39.163424 22895 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:39.165704 22895 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:39.153630 22807 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:39.132668 23202 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:39.140050 23032 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:39.176107 23197 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:39.181811 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:58:39.184105 22774 leader_election.cc:343] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:39.179978 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:39.194181 23197 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:39.195691 23197 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:39.172750 23203 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:39.200088 22848 raft_consensus.cc:359] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:39.200814 22848 raft_consensus.cc:385] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:39.201112 22848 raft_consensus.cc:740] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:39.203876 22849 tablet_bootstrap.cc:654] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:39.204427 22848 consensus_queue.cc:260] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:39.205698 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 72790us
I20251026 14:58:39.196580 23202 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:39.203265 23203 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:39.236799 22848 ts_tablet_manager.cc:1434] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.121s user 0.008s sys 0.001s
I20251026 14:58:39.238332 22848 tablet_bootstrap.cc:492] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:39.243325 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:39.245755 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:39.246347 23032 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:58:39.247300 23266 log.cc:927] Time spent T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
W20251026 14:58:39.252552 23265 log.cc:927] Time spent T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.088s user 0.000s sys 0.001s
W20251026 14:58:39.267263 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 47107us
I20251026 14:58:39.279515 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:58:39.285144 22697 leader_election.cc:343] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:39.286235 22697 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:39.286819 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:39.294332 23196 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:39.294775 23196 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:39.295632 23197 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:39.301952 23196 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
W20251026 14:58:39.311128 22700 leader_election.cc:343] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:39.289336 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:39.325019 22848 tablet_bootstrap.cc:654] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:39.325201 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:39.280560 22624 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:39.318645 22800 raft_consensus.cc:1275] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
W20251026 14:58:39.329566 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 52562.1us
I20251026 14:58:39.330142 23122 raft_consensus.cc:2749] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:39.330192 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:39.328544 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:39.326493 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:39.330221 22732 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:39.342022 22733 raft_consensus.cc:2468] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:39.345505 22728 raft_consensus.cc:2393] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 1: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
W20251026 14:58:39.347483 22700 consensus_peers.cc:597] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e -> Peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Couldn't send request to peer df3a0d1619ee48e8ab68ee0543a72a29. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:39.327863 22803 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 0.
I20251026 14:58:39.348641 22725 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:39.358326 22624 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:39.359499 23206 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:39.359542 23203 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:39.411525 22700 leader_election.cc:343] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:39.413532 22700 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:58:39.417843 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 44559.7us
I20251026 14:58:39.427613 22773 leader_election.cc:304] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:39.419843 23211 raft_consensus.cc:2749] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 14:58:39.428136 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 101629us
I20251026 14:58:39.428871 22773 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:58:39.430490 22774 leader_election.cc:343] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:39.433082 22848 tablet_bootstrap.cc:492] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:39.433604 22848 ts_tablet_manager.cc:1403] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.195s user 0.019s sys 0.000s
I20251026 14:58:39.433775 23196 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:39.434895 22849 tablet_bootstrap.cc:492] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821: No bootstrap required, opened a new log
I20251026 14:58:39.435424 22849 ts_tablet_manager.cc:1403] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821: Time spent bootstrapping tablet: real 0.336s user 0.033s sys 0.011s
I20251026 14:58:39.435765 23196 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:39.423169 22652 raft_consensus.cc:1275] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251026 14:58:39.436046 22848 raft_consensus.cc:359] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:39.438064 22848 raft_consensus.cc:385] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:39.438376 22848 raft_consensus.cc:740] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:39.439173 22848 consensus_queue.cc:260] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:39.449457 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:39.439157 23197 raft_consensus.cc:2804] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
W20251026 14:58:39.453325 22700 leader_election.cc:343] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:39.445955 23218 raft_consensus.cc:2804] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:39.457381 22848 ts_tablet_manager.cc:1434] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20251026 14:58:39.459141 22848 tablet_bootstrap.cc:492] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:39.459348 23218 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:39.462216 23218 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:39.464582 23203 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:39.472016 23197 raft_consensus.cc:697] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:39.481388 23197 consensus_queue.cc:237] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:39.482322 22732 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:39.476083 23272 consensus_queue.cc:1048] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20251026 14:58:39.497956 22659 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
W20251026 14:58:39.501499 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 52640.4us
I20251026 14:58:39.508087 22652 raft_consensus.cc:1275] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251026 14:58:39.514621 22697 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:39.515470 23201 consensus_queue.cc:1048] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 14:58:39.517309 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 78775.6us
I20251026 14:58:39.532050 22773 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:39.534308 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:39.541908 23204 consensus_queue.cc:1048] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:39.543088 23271 raft_consensus.cc:2804] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:39.543407 23204 raft_consensus.cc:2804] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:39.543548 23271 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:39.543933 23204 raft_consensus.cc:697] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:39.547168 22849 raft_consensus.cc:359] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:39.547864 22849 raft_consensus.cc:385] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:39.548132 22849 raft_consensus.cc:740] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Initialized, Role: FOLLOWER
I20251026 14:58:39.523116 23196 consensus_queue.cc:1048] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20251026 14:58:39.548645 22849 consensus_queue.cc:260] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:39.550820 22805 raft_consensus.cc:2393] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 1: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:39.543951 23271 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:39.544832 23204 consensus_queue.cc:237] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:39.561735 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:39.568076 23218 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:39.572235 22849 ts_tablet_manager.cc:1434] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821: Time spent starting tablet: real 0.136s user 0.008s sys 0.002s
I20251026 14:58:39.579161 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:39.580070 22730 raft_consensus.cc:2393] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 1: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:39.599900 22848 tablet_bootstrap.cc:654] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
W20251026 14:58:39.600541 22774 leader_election.cc:343] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:39.607318 22697 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:39.605886 23206 raft_consensus.cc:2749] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:39.612555 23272 raft_consensus.cc:2749] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251026 14:58:39.617937 23122 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:39.620085 23122 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:39.621553 22731 raft_consensus.cc:1275] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251026 14:58:39.616793 23284 consensus_queue.cc:1048] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.058s
I20251026 14:58:39.626277 23122 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:58:39.641062 23286 log.cc:927] Time spent T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
I20251026 14:58:39.642110 23218 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:39.634768 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:58:39.648447 23275 log.cc:927] Time spent T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.312s user 0.001s sys 0.003s
I20251026 14:58:39.649649 23271 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:39.653138 22773 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:39.653695 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:39.656502 23267 raft_consensus.cc:2749] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251026 14:58:39.672207 22622 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:39.681830 22895 raft_consensus.cc:2749] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:39.685359 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:39.685760 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:39.713258 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:39.714573 22803 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:39.724640 23271 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:39.725198 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:39.726064 22733 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
W20251026 14:58:39.750324 22700 leader_election.cc:343] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
W20251026 14:58:39.765542 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 78061.5us
W20251026 14:58:39.769933 22774 leader_election.cc:343] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:39.773944 22656 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:39.783430 23270 consensus_queue.cc:1048] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.095s
I20251026 14:58:39.820814 22733 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:39.830142 22803 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:39.846105 22697 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:39.850099 23282 consensus_queue.cc:1048] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20251026 14:58:39.857649 22773 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:39.862797 22848 tablet_bootstrap.cc:492] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:39.863391 22848 ts_tablet_manager.cc:1403] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.405s user 0.023s sys 0.012s
I20251026 14:58:39.865336 23268 raft_consensus.cc:2804] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:39.865891 23268 raft_consensus.cc:697] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:39.866712 23268 consensus_queue.cc:237] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:39.867772 23298 raft_consensus.cc:2804] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:39.868434 23298 raft_consensus.cc:697] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:39.869241 23298 consensus_queue.cc:237] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:39.929020 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 82288.1us
I20251026 14:58:39.938046 23268 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:39.943830 23268 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:39.959200 23268 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:39.993307 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:39.994519 22732 raft_consensus.cc:2468] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 0.
I20251026 14:58:40.014135 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:40.019168 22848 raft_consensus.cc:359] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:40.020331 22848 raft_consensus.cc:385] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:40.020766 22848 raft_consensus.cc:740] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:40.029516 22773 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:40.032330 23203 raft_consensus.cc:2804] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 14:58:40.032877 23203 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:40.033377 23203 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:40.036595 22848 consensus_queue.cc:260] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:40.039731 22848 ts_tablet_manager.cc:1434] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.176s user 0.010s sys 0.000s
I20251026 14:58:40.039407 23283 consensus_queue.cc:1048] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251026 14:58:40.050474 22848 tablet_bootstrap.cc:492] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
W20251026 14:58:40.110013 23307 log.cc:927] Time spent T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.073s user 0.000s sys 0.001s
I20251026 14:58:40.100647 23203 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:40.121692 23203 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:40.124562 22848 tablet_bootstrap.cc:654] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
W20251026 14:58:40.140043 22774 leader_election.cc:343] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
W20251026 14:58:40.141868 22774 consensus_peers.cc:597] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 -> Peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Couldn't send request to peer df3a0d1619ee48e8ab68ee0543a72a29. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:40.150589 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:40.151564 22725 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:40.156632 23270 consensus_queue.cc:1048] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:40.172271 22725 raft_consensus.cc:2468] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:40.173733 23131 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:40.174371 23131 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:40.180030 23131 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:40.181663 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:40.184680 22773 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:40.190034 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:40.191592 23283 raft_consensus.cc:2804] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 14:58:40.192240 23283 raft_consensus.cc:697] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:40.192904 23283 consensus_queue.cc:237] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:40.197199 22554 catalog_manager.cc:5649] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 0 to 1, leader changed from <none> to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:40.203162 22624 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:40.206642 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:40.209369 23131 raft_consensus.cc:2749] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:40.212435 22848 tablet_bootstrap.cc:492] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:40.216324 22848 ts_tablet_manager.cc:1403] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.166s user 0.035s sys 0.011s
I20251026 14:58:40.220832 22652 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:40.222649 22848 raft_consensus.cc:359] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:40.223615 22848 raft_consensus.cc:385] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:40.224073 22848 raft_consensus.cc:740] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
W20251026 14:58:40.229837 22774 leader_election.cc:343] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Illegal state: must be running to vote when last-logged opid is not known
I20251026 14:58:40.231326 22848 consensus_queue.cc:260] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:40.239185 22848 ts_tablet_manager.cc:1434] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.021s user 0.004s sys 0.003s
I20251026 14:58:40.250437 22848 tablet_bootstrap.cc:492] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:40.277801 22848 tablet_bootstrap.cc:654] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:40.294559 23283 consensus_queue.cc:1048] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:40.299291 22848 tablet_bootstrap.cc:492] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:40.299822 22848 ts_tablet_manager.cc:1403] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.055s user 0.007s sys 0.009s
I20251026 14:58:40.302585 22848 raft_consensus.cc:359] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:40.303282 22848 raft_consensus.cc:385] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:40.303548 22848 raft_consensus.cc:740] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:40.304504 22848 consensus_queue.cc:260] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:40.305898 23206 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:40.306689 23206 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:40.308562 23206 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:40.310057 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:40.311024 22729 raft_consensus.cc:2393] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 1: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:40.312366 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:40.312853 22848 ts_tablet_manager.cc:1434] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20251026 14:58:40.313493 22801 raft_consensus.cc:2393] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 1: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
W20251026 14:58:40.313520 22774 consensus_peers.cc:597] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 -> Peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Couldn't send request to peer df3a0d1619ee48e8ab68ee0543a72a29. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:40.314042 22848 tablet_bootstrap.cc:492] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:40.314711 22622 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:40.315483 23131 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:40.321622 22848 tablet_bootstrap.cc:654] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:40.325781 23131 raft_consensus.cc:2749] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251026 14:58:40.332744 22604 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20251026 14:58:40.333204 22848 tablet_bootstrap.cc:492] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:40.333673 22848 ts_tablet_manager.cc:1403] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.003s
I20251026 14:58:40.335561 22848 raft_consensus.cc:359] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:40.336220 22848 raft_consensus.cc:385] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:40.336462 22848 raft_consensus.cc:740] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:40.337095 22848 consensus_queue.cc:260] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:40.339277 22848 ts_tablet_manager.cc:1434] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251026 14:58:40.341470 22848 tablet_bootstrap.cc:492] T 5373ef47363d446fbf9d697ed507e754 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:40.342139 22654 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:40.349972 22848 tablet_bootstrap.cc:654] T 5373ef47363d446fbf9d697ed507e754 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:40.355618 22848 tablet_bootstrap.cc:492] T 5373ef47363d446fbf9d697ed507e754 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:40.355558 23258 consensus_queue.cc:1048] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:40.356235 22848 ts_tablet_manager.cc:1403] T 5373ef47363d446fbf9d697ed507e754 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20251026 14:58:40.358844 22848 raft_consensus.cc:359] T 5373ef47363d446fbf9d697ed507e754 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:40.359542 22848 raft_consensus.cc:385] T 5373ef47363d446fbf9d697ed507e754 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:40.360152 22848 raft_consensus.cc:740] T 5373ef47363d446fbf9d697ed507e754 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:40.361340 22848 consensus_queue.cc:260] T 5373ef47363d446fbf9d697ed507e754 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:40.364483 22848 ts_tablet_manager.cc:1434] T 5373ef47363d446fbf9d697ed507e754 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20251026 14:58:40.365504 22848 tablet_bootstrap.cc:492] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:40.371650 22848 tablet_bootstrap.cc:654] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:40.379334 23202 consensus_queue.cc:1048] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 14:58:40.398243 22552 catalog_manager.cc:5649] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:40.400750 22552 catalog_manager.cc:5649] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:40.407553 22552 catalog_manager.cc:5649] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:40.413040 22848 tablet_bootstrap.cc:492] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:40.413589 22848 ts_tablet_manager.cc:1403] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.048s user 0.018s sys 0.025s
I20251026 14:58:40.416181 22848 raft_consensus.cc:359] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:40.416817 22848 raft_consensus.cc:385] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:40.417115 22848 raft_consensus.cc:740] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:40.417838 22848 consensus_queue.cc:260] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:40.421351 22848 ts_tablet_manager.cc:1434] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20251026 14:58:40.422394 22848 tablet_bootstrap.cc:492] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29: Bootstrap starting.
I20251026 14:58:40.428287 22848 tablet_bootstrap.cc:654] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29: Neither blocks nor log segments found. Creating new log.
I20251026 14:58:40.440078 22848 tablet_bootstrap.cc:492] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29: No bootstrap required, opened a new log
I20251026 14:58:40.440996 22848 ts_tablet_manager.cc:1403] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.008s
I20251026 14:58:40.444849 22848 raft_consensus.cc:359] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:40.445674 22848 raft_consensus.cc:385] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 14:58:40.445997 22848 raft_consensus.cc:740] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Initialized, Role: FOLLOWER
I20251026 14:58:40.446985 22848 consensus_queue.cc:260] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:40.449491 22848 ts_tablet_manager.cc:1434] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20251026 14:58:40.465099 22554 catalog_manager.cc:5649] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:40.476598 23268 consensus_queue.cc:1048] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:40.483732 22654 raft_consensus.cc:3060] T 5373ef47363d446fbf9d697ed507e754 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:40.639233 22654 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:40.647143 23203 consensus_queue.cc:1048] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:40.657953 23268 consensus_queue.cc:1048] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:40.759589 23131 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:40.760200 23131 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:40.762168 23131 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:40.762815 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:40.763670 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:40.765547 22622 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:40.766252 23131 raft_consensus.cc:2749] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 14:58:40.838142 22654 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:40.946560 22654 raft_consensus.cc:3060] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 0 FOLLOWER]: Advancing to term 1
I20251026 14:58:41.212795 23300 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:41.213711 23300 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:41.216887 23300 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:41.217583 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:41.217712 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:41.218266 22657 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:41.218281 22801 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:41.219444 22700 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:41.220098 23300 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:41.220422 23300 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:41.220639 23300 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:41.226854 23300 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:41.228878 23300 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:41.229611 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:41.229481 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:41.230225 22801 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:41.230448 22657 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:41.236546 22801 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:41.237787 22697 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:41.238390 23300 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:41.238677 22657 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:41.238765 23300 raft_consensus.cc:697] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 2 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:41.239789 23300 consensus_queue.cc:237] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:41.252938 22554 catalog_manager.cc:5649] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 0 to 2, leader changed from <none> to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 2 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:41.836539 22654 raft_consensus.cc:1275] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251026 14:58:41.856392 23351 consensus_queue.cc:1048] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:41.914690 22805 raft_consensus.cc:1275] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251026 14:58:41.930807 23202 consensus_queue.cc:1048] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 14:58:42.547464 21512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 14:58:42.575167 23361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 14:58:42.575198 23358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 14:58:42.576150 23359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 14:58:42.578228 21512 server_base.cc:1047] running on GCE node
I20251026 14:58:42.582276 21512 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 14:58:42.582648 21512 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 14:58:42.582828 21512 hybrid_clock.cc:648] HybridClock initialized: now 1761490722582811 us; error 0 us; skew 500 ppm
I20251026 14:58:42.583400 21512 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 14:58:42.586452 21512 webserver.cc:492] Webserver started at http://127.21.2.4:38641/ using document root <none> and password file <none>
I20251026 14:58:42.586951 21512 fs_manager.cc:362] Metadata directory not provided
I20251026 14:58:42.587165 21512 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 14:58:42.587445 21512 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 14:58:42.588678 21512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-3-root/instance:
uuid: "248493ec312e4398b828c6cb00e5c4b1"
format_stamp: "Formatted at 2025-10-26 14:58:42 on dist-test-slave-9n4f"
I20251026 14:58:42.594156 21512 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20251026 14:58:42.597579 23366 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 14:58:42.598421 21512 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251026 14:58:42.598713 21512 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-3-root
uuid: "248493ec312e4398b828c6cb00e5c4b1"
format_stamp: "Formatted at 2025-10-26 14:58:42 on dist-test-slave-9n4f"
I20251026 14:58:42.598966 21512 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 14:58:42.631667 21512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 14:58:42.633267 21512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 14:58:42.635851 21512 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 14:58:42.639160 21512 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 14:58:42.639443 21512 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 14:58:42.639758 21512 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 14:58:42.639961 21512 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 14:58:42.692763 21512 rpc_server.cc:307] RPC server started. Bound to: 127.21.2.4:43341
I20251026 14:58:42.693508 23428 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.2.4:43341 every 8 connection(s)
I20251026 14:58:42.706315 21512 auto_leader_rebalancer-test.cc:417] add a tserver: 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:58:42.715817 23429 heartbeater.cc:344] Connected to a master server at 127.21.2.62:38611
I20251026 14:58:42.716305 23429 heartbeater.cc:461] Registering TS with master...
I20251026 14:58:42.717235 23429 heartbeater.cc:507] Master 127.21.2.62:38611 requested a full tablet report, sending...
W20251026 14:58:42.718468 22755 debug-util.cc:398] Leaking SignalData structure 0x7b0800167ce0 after lost signal to thread 21515
I20251026 14:58:42.719458 22552 ts_manager.cc:194] Registered new tserver with Master: 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
W20251026 14:58:42.720326 22755 debug-util.cc:398] Leaking SignalData structure 0x7b08002c2320 after lost signal to thread 22683
W20251026 14:58:42.721413 22755 debug-util.cc:398] Leaking SignalData structure 0x7b08002d6d20 after lost signal to thread 22758
I20251026 14:58:42.721357 22552 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50170
W20251026 14:58:42.722554 22755 debug-util.cc:398] Leaking SignalData structure 0x7b080028d960 after lost signal to thread 22832
I20251026 14:58:43.528501 22808 consensus_queue.cc:237] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:43.537840 22726 raft_consensus.cc:1275] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 14:58:43.539798 22654 raft_consensus.cc:1275] T 5373ef47363d446fbf9d697ed507e754 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 14:58:43.539615 23433 consensus_queue.cc:1048] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:43.542253 23433 consensus_queue.cc:1048] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251026 14:58:43.555655 22771 consensus_peers.cc:597] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 -> Peer 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341): Couldn't send request to peer 248493ec312e4398b828c6cb00e5c4b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5373ef47363d446fbf9d697ed507e754. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:43.558295 23436 raft_consensus.cc:2955] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:43.562222 22654 raft_consensus.cc:2955] T 5373ef47363d446fbf9d697ed507e754 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:43.565781 22726 raft_consensus.cc:2955] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:43.586265 22552 catalog_manager.cc:5649] T 5373ef47363d446fbf9d697ed507e754 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
W20251026 14:58:43.638453 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 67023.9us
I20251026 14:58:43.650564 22726 consensus_queue.cc:237] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:43.724525 23429 heartbeater.cc:499] Master 127.21.2.62:38611 was elected leader, sending a full tablet report...
I20251026 14:58:43.749763 22654 raft_consensus.cc:1275] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 14:58:43.751415 23202 consensus_queue.cc:1048] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:43.763464 23202 raft_consensus.cc:2955] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:43.767123 22651 raft_consensus.cc:2955] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:43.786742 22805 consensus_queue.cc:237] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:43.791580 22553 catalog_manager.cc:5649] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:43.807191 22726 raft_consensus.cc:1275] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251026 14:58:43.808413 22771 consensus_peers.cc:597] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 -> Peer 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341): Couldn't send request to peer 248493ec312e4398b828c6cb00e5c4b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a0e9c0787aa4053afa2791a86bb263a. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:43.809026 23433 consensus_queue.cc:1048] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:58:43.812530 22654 raft_consensus.cc:1275] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 14:58:43.814105 23433 consensus_queue.cc:1048] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:43.823905 22805 raft_consensus.cc:1275] T a673ec034b704e8e86556afe9322d8c2 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 14:58:43.828778 23455 consensus_queue.cc:1048] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:58:43.835311 23433 raft_consensus.cc:2955] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:43.838775 22726 raft_consensus.cc:2955] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:43.839707 22654 raft_consensus.cc:2955] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:43.856392 22805 raft_consensus.cc:2955] T a673ec034b704e8e86556afe9322d8c2 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:43.871935 22734 consensus_queue.cc:237] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:43.876430 22553 catalog_manager.cc:5649] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
W20251026 14:58:43.894112 22697 consensus_peers.cc:597] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e -> Peer 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341): Couldn't send request to peer 248493ec312e4398b828c6cb00e5c4b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a673ec034b704e8e86556afe9322d8c2. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:43.911548 22651 raft_consensus.cc:1275] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 14:58:43.913126 23202 consensus_queue.cc:1048] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251026 14:58:43.916991 22697 consensus_peers.cc:597] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e -> Peer 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341): Couldn't send request to peer 248493ec312e4398b828c6cb00e5c4b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aceef9b41bab4ad3bf67b4d597a7fe11. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:43.918337 22805 raft_consensus.cc:1275] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 14:58:43.919801 23202 consensus_queue.cc:1048] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:43.950373 23202 raft_consensus.cc:2955] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:43.953835 22651 raft_consensus.cc:2955] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:43.969496 23465 ts_tablet_manager.cc:933] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1: Initiating tablet copy from peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:43.996496 23465 tablet_copy_client.cc:323] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.2.3:39645
I20251026 14:58:44.023406 22818 tablet_copy_service.cc:140] P 0d684445287b450a8f738b6ad4b59821: Received BeginTabletCopySession request for tablet 5373ef47363d446fbf9d697ed507e754 from peer 248493ec312e4398b828c6cb00e5c4b1 ({username='slave'} at 127.0.0.1:49652)
I20251026 14:58:44.024158 22818 tablet_copy_service.cc:161] P 0d684445287b450a8f738b6ad4b59821: Beginning new tablet copy session on tablet 5373ef47363d446fbf9d697ed507e754 from peer 248493ec312e4398b828c6cb00e5c4b1 at {username='slave'} at 127.0.0.1:49652: session id = 248493ec312e4398b828c6cb00e5c4b1-5373ef47363d446fbf9d697ed507e754
I20251026 14:58:44.039184 22818 tablet_copy_source_session.cc:215] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 14:58:44.040313 22553 catalog_manager.cc:5649] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:44.041209 22800 raft_consensus.cc:2955] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:44.066403 23465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5373ef47363d446fbf9d697ed507e754. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:44.090301 23465 tablet_copy_client.cc:806] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Starting download of 0 data blocks...
I20251026 14:58:44.091060 23465 tablet_copy_client.cc:670] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Starting download of 1 WAL segments...
I20251026 14:58:44.096266 23465 tablet_copy_client.cc:538] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 14:58:44.107847 23465 tablet_bootstrap.cc:492] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap starting.
I20251026 14:58:44.150887 23465 tablet_bootstrap.cc:492] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 14:58:44.152125 23465 tablet_bootstrap.cc:492] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap complete.
I20251026 14:58:44.152964 23465 ts_tablet_manager.cc:1403] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1: Time spent bootstrapping tablet: real 0.045s user 0.032s sys 0.013s
I20251026 14:58:44.156513 23465 raft_consensus.cc:359] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:44.157608 23465 raft_consensus.cc:740] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Initialized, Role: LEARNER
I20251026 14:58:44.158386 23465 consensus_queue.cc:260] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:44.206125 23471 ts_tablet_manager.cc:933] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1: Initiating tablet copy from peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:44.210668 23465 ts_tablet_manager.cc:1434] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1: Time spent starting tablet: real 0.057s user 0.016s sys 0.034s
I20251026 14:58:44.212846 22818 tablet_copy_service.cc:342] P 0d684445287b450a8f738b6ad4b59821: Request end of tablet copy session 248493ec312e4398b828c6cb00e5c4b1-5373ef47363d446fbf9d697ed507e754 received from {username='slave'} at 127.0.0.1:49652
I20251026 14:58:44.213445 22818 tablet_copy_service.cc:434] P 0d684445287b450a8f738b6ad4b59821: ending tablet copy session 248493ec312e4398b828c6cb00e5c4b1-5373ef47363d446fbf9d697ed507e754 on tablet 5373ef47363d446fbf9d697ed507e754 with peer 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:58:44.222265 23471 tablet_copy_client.cc:323] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.2.2:46507
I20251026 14:58:44.244099 22744 tablet_copy_service.cc:140] P 562bb1e1c8e445b8baecd29c3669658e: Received BeginTabletCopySession request for tablet a673ec034b704e8e86556afe9322d8c2 from peer 248493ec312e4398b828c6cb00e5c4b1 ({username='slave'} at 127.0.0.1:47996)
I20251026 14:58:44.244755 22744 tablet_copy_service.cc:161] P 562bb1e1c8e445b8baecd29c3669658e: Beginning new tablet copy session on tablet a673ec034b704e8e86556afe9322d8c2 from peer 248493ec312e4398b828c6cb00e5c4b1 at {username='slave'} at 127.0.0.1:47996: session id = 248493ec312e4398b828c6cb00e5c4b1-a673ec034b704e8e86556afe9322d8c2
I20251026 14:58:44.252230 23465 ts_tablet_manager.cc:933] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1: Initiating tablet copy from peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:44.263739 22744 tablet_copy_source_session.cc:215] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 14:58:44.271607 23465 tablet_copy_client.cc:323] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.2.3:39645
I20251026 14:58:44.272642 23471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a673ec034b704e8e86556afe9322d8c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:44.273547 22818 tablet_copy_service.cc:140] P 0d684445287b450a8f738b6ad4b59821: Received BeginTabletCopySession request for tablet 5a0e9c0787aa4053afa2791a86bb263a from peer 248493ec312e4398b828c6cb00e5c4b1 ({username='slave'} at 127.0.0.1:49652)
I20251026 14:58:44.274096 22818 tablet_copy_service.cc:161] P 0d684445287b450a8f738b6ad4b59821: Beginning new tablet copy session on tablet 5a0e9c0787aa4053afa2791a86bb263a from peer 248493ec312e4398b828c6cb00e5c4b1 at {username='slave'} at 127.0.0.1:49652: session id = 248493ec312e4398b828c6cb00e5c4b1-5a0e9c0787aa4053afa2791a86bb263a
I20251026 14:58:44.281175 22818 tablet_copy_source_session.cc:215] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 14:58:44.284757 23465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a0e9c0787aa4053afa2791a86bb263a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:44.317505 23471 tablet_copy_client.cc:806] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Starting download of 0 data blocks...
I20251026 14:58:44.318132 23471 tablet_copy_client.cc:670] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Starting download of 1 WAL segments...
I20251026 14:58:44.318727 23465 tablet_copy_client.cc:806] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Starting download of 0 data blocks...
I20251026 14:58:44.335659 23465 tablet_copy_client.cc:670] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Starting download of 1 WAL segments...
I20251026 14:58:44.340368 23471 tablet_copy_client.cc:538] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 14:58:44.348649 23465 tablet_copy_client.cc:538] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 14:58:44.350790 23404 raft_consensus.cc:1217] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251026 14:58:44.352564 23471 tablet_bootstrap.cc:492] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap starting.
I20251026 14:58:44.359532 23465 tablet_bootstrap.cc:492] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap starting.
I20251026 14:58:44.389741 23465 tablet_bootstrap.cc:492] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 14:58:44.390988 23465 tablet_bootstrap.cc:492] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap complete.
I20251026 14:58:44.391974 23465 ts_tablet_manager.cc:1403] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1: Time spent bootstrapping tablet: real 0.033s user 0.023s sys 0.007s
I20251026 14:58:44.395129 23465 raft_consensus.cc:359] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:44.396180 23465 raft_consensus.cc:740] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Initialized, Role: LEARNER
I20251026 14:58:44.403169 23465 consensus_queue.cc:260] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:44.409593 23465 ts_tablet_manager.cc:1434] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20251026 14:58:44.411490 22818 tablet_copy_service.cc:342] P 0d684445287b450a8f738b6ad4b59821: Request end of tablet copy session 248493ec312e4398b828c6cb00e5c4b1-5a0e9c0787aa4053afa2791a86bb263a received from {username='slave'} at 127.0.0.1:49652
I20251026 14:58:44.411972 22818 tablet_copy_service.cc:434] P 0d684445287b450a8f738b6ad4b59821: ending tablet copy session 248493ec312e4398b828c6cb00e5c4b1-5a0e9c0787aa4053afa2791a86bb263a on tablet 5a0e9c0787aa4053afa2791a86bb263a with peer 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:58:44.435051 23471 tablet_bootstrap.cc:492] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 14:58:44.436118 23471 tablet_bootstrap.cc:492] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap complete.
I20251026 14:58:44.437335 23471 ts_tablet_manager.cc:1403] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1: Time spent bootstrapping tablet: real 0.085s user 0.032s sys 0.012s
I20251026 14:58:44.440007 23471 raft_consensus.cc:359] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:44.441718 23471 raft_consensus.cc:740] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Initialized, Role: LEARNER
I20251026 14:58:44.442386 23471 consensus_queue.cc:260] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:44.443732 23465 ts_tablet_manager.cc:933] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1: Initiating tablet copy from peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:44.453428 23471 ts_tablet_manager.cc:1434] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1: Time spent starting tablet: real 0.016s user 0.004s sys 0.003s
I20251026 14:58:44.455468 23465 tablet_copy_client.cc:323] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.2.2:46507
I20251026 14:58:44.458472 22744 tablet_copy_service.cc:342] P 562bb1e1c8e445b8baecd29c3669658e: Request end of tablet copy session 248493ec312e4398b828c6cb00e5c4b1-a673ec034b704e8e86556afe9322d8c2 received from {username='slave'} at 127.0.0.1:47996
I20251026 14:58:44.459036 22744 tablet_copy_service.cc:434] P 562bb1e1c8e445b8baecd29c3669658e: ending tablet copy session 248493ec312e4398b828c6cb00e5c4b1-a673ec034b704e8e86556afe9322d8c2 on tablet a673ec034b704e8e86556afe9322d8c2 with peer 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:58:44.463486 22744 tablet_copy_service.cc:140] P 562bb1e1c8e445b8baecd29c3669658e: Received BeginTabletCopySession request for tablet aceef9b41bab4ad3bf67b4d597a7fe11 from peer 248493ec312e4398b828c6cb00e5c4b1 ({username='slave'} at 127.0.0.1:47996)
I20251026 14:58:44.464130 22744 tablet_copy_service.cc:161] P 562bb1e1c8e445b8baecd29c3669658e: Beginning new tablet copy session on tablet aceef9b41bab4ad3bf67b4d597a7fe11 from peer 248493ec312e4398b828c6cb00e5c4b1 at {username='slave'} at 127.0.0.1:47996: session id = 248493ec312e4398b828c6cb00e5c4b1-aceef9b41bab4ad3bf67b4d597a7fe11
I20251026 14:58:44.477144 22744 tablet_copy_source_session.cc:215] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 14:58:44.480175 23465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aceef9b41bab4ad3bf67b4d597a7fe11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:44.503775 23465 tablet_copy_client.cc:806] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Starting download of 0 data blocks...
I20251026 14:58:44.504505 23465 tablet_copy_client.cc:670] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Starting download of 1 WAL segments...
I20251026 14:58:44.515236 23465 tablet_copy_client.cc:538] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 14:58:44.532948 23465 tablet_bootstrap.cc:492] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap starting.
I20251026 14:58:44.567443 23465 tablet_bootstrap.cc:492] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 14:58:44.568603 23465 tablet_bootstrap.cc:492] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap complete.
I20251026 14:58:44.569391 23465 ts_tablet_manager.cc:1403] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1: Time spent bootstrapping tablet: real 0.037s user 0.021s sys 0.009s
I20251026 14:58:44.573207 23465 raft_consensus.cc:359] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:44.574234 23465 raft_consensus.cc:740] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Initialized, Role: LEARNER
I20251026 14:58:44.574960 23465 consensus_queue.cc:260] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:44.577440 23465 ts_tablet_manager.cc:1434] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20251026 14:58:44.578961 22744 tablet_copy_service.cc:342] P 562bb1e1c8e445b8baecd29c3669658e: Request end of tablet copy session 248493ec312e4398b828c6cb00e5c4b1-aceef9b41bab4ad3bf67b4d597a7fe11 received from {username='slave'} at 127.0.0.1:47996
I20251026 14:58:44.579413 22744 tablet_copy_service.cc:434] P 562bb1e1c8e445b8baecd29c3669658e: ending tablet copy session 248493ec312e4398b828c6cb00e5c4b1-aceef9b41bab4ad3bf67b4d597a7fe11 on tablet aceef9b41bab4ad3bf67b4d597a7fe11 with peer 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:58:44.594844 23404 raft_consensus.cc:1217] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251026 14:58:44.837971 23404 raft_consensus.cc:1217] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251026 14:58:44.873530 23404 raft_consensus.cc:1217] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251026 14:58:44.890271 23448 raft_consensus.cc:1064] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821: attempting to promote NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 to VOTER
I20251026 14:58:44.896474 23448 consensus_queue.cc:237] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:44.923071 22730 raft_consensus.cc:1275] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:44.923230 22651 raft_consensus.cc:1275] T 5373ef47363d446fbf9d697ed507e754 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:44.924782 23448 consensus_queue.cc:1048] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:44.926887 23404 raft_consensus.cc:1275] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:44.931198 23446 consensus_queue.cc:1048] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:58:44.932588 23436 consensus_queue.cc:1048] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:44.983501 23448 raft_consensus.cc:2955] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:44.988135 23404 raft_consensus.cc:2955] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:44.991183 22651 raft_consensus.cc:2955] T 5373ef47363d446fbf9d697ed507e754 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.000212 22726 raft_consensus.cc:2955] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.012645 22553 catalog_manager.cc:5649] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.071812 22801 consensus_queue.cc:237] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:45.079363 23202 raft_consensus.cc:1064] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: attempting to promote NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 to VOTER
I20251026 14:58:45.081696 23404 raft_consensus.cc:1275] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 14:58:45.084913 23448 consensus_queue.cc:1048] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251026 14:58:45.087807 22730 raft_consensus.cc:1275] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 14:58:45.087538 23202 consensus_queue.cc:237] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:45.092892 23446 raft_consensus.cc:2955] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.097136 23436 consensus_queue.cc:1048] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251026 14:58:45.105892 23404 raft_consensus.cc:2955] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.111734 23403 raft_consensus.cc:1275] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:45.152891 22540 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 5373ef47363d446fbf9d697ed507e754 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 14:58:45.155589 23202 consensus_queue.cc:1048] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:45.160975 22554 catalog_manager.cc:5649] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: config changed from index 3 to 4, VOTER df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) evicted. New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
W20251026 14:58:45.166010 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 68368us
W20251026 14:58:45.187772 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 74408.9us
I20251026 14:58:45.190013 22640 tablet_service.cc:1552] Processing DeleteTablet for tablet 5373ef47363d446fbf9d697ed507e754 with delete_type TABLET_DATA_TOMBSTONED (TS df3a0d1619ee48e8ab68ee0543a72a29 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54560
W20251026 14:58:45.177593 23487 log.cc:927] Time spent T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.085s user 0.003s sys 0.012s
W20251026 14:58:45.204452 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 97111.3us
I20251026 14:58:45.207463 22651 raft_consensus.cc:1275] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:45.212142 22808 raft_consensus.cc:1275] T a673ec034b704e8e86556afe9322d8c2 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:45.221874 22730 raft_consensus.cc:2955] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.223802 23467 consensus_queue.cc:1048] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:58:45.223801 23493 tablet_replica.cc:333] T 5373ef47363d446fbf9d697ed507e754 P df3a0d1619ee48e8ab68ee0543a72a29: stopping tablet replica
I20251026 14:58:45.230533 23493 raft_consensus.cc:2243] T 5373ef47363d446fbf9d697ed507e754 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:45.231667 23493 raft_consensus.cc:2272] T 5373ef47363d446fbf9d697ed507e754 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:45.235057 23493 ts_tablet_manager.cc:1916] T 5373ef47363d446fbf9d697ed507e754 P df3a0d1619ee48e8ab68ee0543a72a29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 14:58:45.250715 23467 raft_consensus.cc:1064] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e: attempting to promote NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 to VOTER
I20251026 14:58:45.253681 23467 consensus_queue.cc:237] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:45.256697 23493 ts_tablet_manager.cc:1929] T 5373ef47363d446fbf9d697ed507e754 P df3a0d1619ee48e8ab68ee0543a72a29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251026 14:58:45.257432 23493 log.cc:1199] T 5373ef47363d446fbf9d697ed507e754 P df3a0d1619ee48e8ab68ee0543a72a29: Deleting WAL directory at /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-0-root/wals/5373ef47363d446fbf9d697ed507e754
I20251026 14:58:45.259606 22543 catalog_manager.cc:4985] TS df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): tablet 5373ef47363d446fbf9d697ed507e754 (table test-workload [id=0253264dd7b746d58adac3c9eeddb55d]) successfully deleted
I20251026 14:58:45.274720 23202 raft_consensus.cc:2955] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.290693 23490 consensus_queue.cc:1048] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:58:45.294240 22808 raft_consensus.cc:2955] T a673ec034b704e8e86556afe9322d8c2 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.297438 23404 raft_consensus.cc:1275] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:45.299449 23491 consensus_queue.cc:1048] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:58:45.304085 22801 raft_consensus.cc:1275] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:45.306322 23490 consensus_queue.cc:1048] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:45.322409 22654 raft_consensus.cc:1275] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:45.325805 23466 consensus_queue.cc:1048] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:58:45.328140 22651 raft_consensus.cc:2955] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.341527 23404 raft_consensus.cc:2955] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.351044 22554 catalog_manager.cc:5649] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.391736 23467 raft_consensus.cc:2955] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.402813 22801 raft_consensus.cc:2955] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.424279 23404 raft_consensus.cc:2955] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.481176 22654 raft_consensus.cc:2955] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.481712 22554 catalog_manager.cc:5649] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.488109 23489 raft_consensus.cc:1064] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821: attempting to promote NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 to VOTER
I20251026 14:58:45.490787 23489 consensus_queue.cc:237] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:45.506624 23404 raft_consensus.cc:1275] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:45.508852 23436 consensus_queue.cc:1048] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:45.509895 22729 raft_consensus.cc:1275] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:45.511780 22652 raft_consensus.cc:1275] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:45.515003 23489 consensus_queue.cc:1048] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:45.515959 23448 consensus_queue.cc:1048] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:45.566891 22725 consensus_queue.cc:237] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:45.581223 23404 raft_consensus.cc:1275] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 14:58:45.586836 23466 consensus_queue.cc:1048] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:58:45.590452 23436 raft_consensus.cc:2955] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.593608 22652 raft_consensus.cc:2955] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.595844 23491 raft_consensus.cc:2955] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.604118 23403 raft_consensus.cc:2955] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.610397 23404 raft_consensus.cc:2955] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.632449 22553 catalog_manager.cc:5649] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.641790 22652 raft_consensus.cc:1275] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 14:58:45.648342 22542 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet aceef9b41bab4ad3bf67b4d597a7fe11 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 14:58:45.656821 22801 consensus_queue.cc:237] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:45.654057 22730 raft_consensus.cc:2955] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.644440 23466 consensus_queue.cc:1048] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20251026 14:58:45.685932 22555 catalog_manager.cc:5649] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: config changed from index 3 to 4, VOTER 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) evicted. New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.690980 23404 raft_consensus.cc:1275] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 14:58:45.704836 22791 tablet_service.cc:1552] Processing DeleteTablet for tablet aceef9b41bab4ad3bf67b4d597a7fe11 with delete_type TABLET_DATA_TOMBSTONED (TS 0d684445287b450a8f738b6ad4b59821 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44570
I20251026 14:58:45.706940 23436 consensus_queue.cc:1048] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251026 14:58:45.708905 22651 raft_consensus.cc:1275] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 14:58:45.713428 23506 tablet_replica.cc:333] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821: stopping tablet replica
I20251026 14:58:45.717990 23506 raft_consensus.cc:2243] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:45.718775 23506 raft_consensus.cc:2272] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:45.721998 23489 raft_consensus.cc:2955] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.723877 23404 raft_consensus.cc:2955] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.730954 23506 ts_tablet_manager.cc:1916] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 14:58:45.740887 22651 raft_consensus.cc:2955] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.747185 22553 catalog_manager.cc:5649] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: config changed from index 3 to 4, VOTER 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) evicted. New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.753611 22540 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 5a0e9c0787aa4053afa2791a86bb263a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 14:58:45.788050 23446 consensus_queue.cc:1048] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:58:45.790213 22720 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a0e9c0787aa4053afa2791a86bb263a with delete_type TABLET_DATA_TOMBSTONED (TS 562bb1e1c8e445b8baecd29c3669658e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35078
I20251026 14:58:45.799158 23508 tablet_replica.cc:333] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e: stopping tablet replica
I20251026 14:58:45.800501 23508 raft_consensus.cc:2243] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:45.801456 23508 raft_consensus.cc:2272] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:45.804991 23508 ts_tablet_manager.cc:1916] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 14:58:45.811635 22651 raft_consensus.cc:2955] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.847517 22729 consensus_queue.cc:237] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:45.855726 23404 raft_consensus.cc:1275] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 14:58:45.857555 23492 consensus_queue.cc:1048] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251026 14:58:45.859217 23508 ts_tablet_manager.cc:1929] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251026 14:58:45.859706 23508 log.cc:1199] T 5a0e9c0787aa4053afa2791a86bb263a P 562bb1e1c8e445b8baecd29c3669658e: Deleting WAL directory at /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-1-root/wals/5a0e9c0787aa4053afa2791a86bb263a
I20251026 14:58:45.861202 22652 raft_consensus.cc:1275] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 14:58:45.866063 23506 ts_tablet_manager.cc:1929] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251026 14:58:45.866519 23506 log.cc:1199] T aceef9b41bab4ad3bf67b4d597a7fe11 P 0d684445287b450a8f738b6ad4b59821: Deleting WAL directory at /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-2-root/wals/aceef9b41bab4ad3bf67b4d597a7fe11
I20251026 14:58:45.868265 22540 catalog_manager.cc:4985] TS 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): tablet aceef9b41bab4ad3bf67b4d597a7fe11 (table test-workload [id=0253264dd7b746d58adac3c9eeddb55d]) successfully deleted
I20251026 14:58:45.870751 23202 consensus_queue.cc:1048] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:58:45.870090 23507 raft_consensus.cc:2955] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.871158 23404 raft_consensus.cc:2955] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.875257 22652 raft_consensus.cc:2955] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.885344 22542 catalog_manager.cc:4985] TS 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): tablet 5a0e9c0787aa4053afa2791a86bb263a (table test-workload [id=0253264dd7b746d58adac3c9eeddb55d]) successfully deleted
I20251026 14:58:45.892441 22542 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet a673ec034b704e8e86556afe9322d8c2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 14:58:45.898381 22555 catalog_manager.cc:5649] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: config changed from index 3 to 4, VOTER 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) evicted. New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:45.910220 22791 tablet_service.cc:1552] Processing DeleteTablet for tablet a673ec034b704e8e86556afe9322d8c2 with delete_type TABLET_DATA_TOMBSTONED (TS 0d684445287b450a8f738b6ad4b59821 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44570
I20251026 14:58:45.911275 23506 tablet_replica.cc:333] T a673ec034b704e8e86556afe9322d8c2 P 0d684445287b450a8f738b6ad4b59821: stopping tablet replica
I20251026 14:58:45.913020 23506 raft_consensus.cc:2243] T a673ec034b704e8e86556afe9322d8c2 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:45.914254 23506 raft_consensus.cc:2272] T a673ec034b704e8e86556afe9322d8c2 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:45.917814 23506 ts_tablet_manager.cc:1916] T a673ec034b704e8e86556afe9322d8c2 P 0d684445287b450a8f738b6ad4b59821: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 14:58:45.940205 23506 ts_tablet_manager.cc:1929] T a673ec034b704e8e86556afe9322d8c2 P 0d684445287b450a8f738b6ad4b59821: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251026 14:58:45.940675 23506 log.cc:1199] T a673ec034b704e8e86556afe9322d8c2 P 0d684445287b450a8f738b6ad4b59821: Deleting WAL directory at /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-2-root/wals/a673ec034b704e8e86556afe9322d8c2
I20251026 14:58:45.942376 22540 catalog_manager.cc:4985] TS 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): tablet a673ec034b704e8e86556afe9322d8c2 (table test-workload [id=0253264dd7b746d58adac3c9eeddb55d]) successfully deleted
I20251026 14:58:46.994341 22652 consensus_queue.cc:237] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:47.002802 22730 raft_consensus.cc:1275] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 14:58:47.003011 22808 raft_consensus.cc:1275] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 14:58:47.004554 23512 consensus_queue.cc:1048] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:58:47.006419 23435 consensus_queue.cc:1048] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:47.014317 23512 raft_consensus.cc:2955] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:47.016717 22808 raft_consensus.cc:2955] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
W20251026 14:58:47.019645 22622 consensus_peers.cc:597] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 -> Peer 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341): Couldn't send request to peer 248493ec312e4398b828c6cb00e5c4b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 741e08995c924c8e9ea37c02c41a5bc7. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:47.020761 22725 raft_consensus.cc:2955] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:47.030351 22805 consensus_queue.cc:237] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:47.034055 22552 catalog_manager.cc:5649] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New cstate: current_term: 1 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251026 14:58:47.057543 22771 consensus_peers.cc:597] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 -> Peer 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341): Couldn't send request to peer 248493ec312e4398b828c6cb00e5c4b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02799da19b36403499a63101b8418ffe. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:47.057941 22725 raft_consensus.cc:1275] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 14:58:47.058406 22654 raft_consensus.cc:1275] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 14:58:47.059674 23489 consensus_queue.cc:1048] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:47.060931 23446 consensus_queue.cc:1048] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:47.087103 23446 raft_consensus.cc:2955] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:47.093107 22652 raft_consensus.cc:2955] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:47.095166 22725 raft_consensus.cc:2955] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:47.111549 22555 catalog_manager.cc:5649] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 reported cstate change: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:47.116704 22654 consensus_queue.cc:237] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:47.141729 22725 raft_consensus.cc:1275] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251026 14:58:47.142983 22622 consensus_peers.cc:597] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 -> Peer 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341): Couldn't send request to peer 248493ec312e4398b828c6cb00e5c4b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a35eb71706aa4f97bf6c64430916aef5. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:47.143532 22805 raft_consensus.cc:1275] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 14:58:47.145293 23435 consensus_queue.cc:1048] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:58:47.147384 23435 consensus_queue.cc:1048] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:47.155776 23501 raft_consensus.cc:2955] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:47.160037 22808 raft_consensus.cc:2955] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:47.159677 22725 raft_consensus.cc:2955] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:47.173655 22808 consensus_queue.cc:237] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:47.180325 22555 catalog_manager.cc:5649] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New cstate: current_term: 1 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
W20251026 14:58:47.209939 22771 consensus_peers.cc:597] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 -> Peer 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341): Couldn't send request to peer 248493ec312e4398b828c6cb00e5c4b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea1e7da57b65433b9e01cad8c4adeb75. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:47.211169 22725 raft_consensus.cc:1275] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 14:58:47.212339 23436 consensus_queue.cc:1048] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:47.213699 22652 raft_consensus.cc:1275] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 14:58:47.215328 23436 consensus_queue.cc:1048] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:47.223687 23489 raft_consensus.cc:2955] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:47.225504 22725 raft_consensus.cc:2955] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:47.226246 22652 raft_consensus.cc:2955] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:47.242507 22552 catalog_manager.cc:5649] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:47.582613 23532 ts_tablet_manager.cc:933] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1: Initiating tablet copy from peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:58:47.585765 23532 tablet_copy_client.cc:323] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.2.1:46185
I20251026 14:58:47.598661 22669 tablet_copy_service.cc:140] P df3a0d1619ee48e8ab68ee0543a72a29: Received BeginTabletCopySession request for tablet 741e08995c924c8e9ea37c02c41a5bc7 from peer 248493ec312e4398b828c6cb00e5c4b1 ({username='slave'} at 127.0.0.1:32888)
I20251026 14:58:47.599243 22669 tablet_copy_service.cc:161] P df3a0d1619ee48e8ab68ee0543a72a29: Beginning new tablet copy session on tablet 741e08995c924c8e9ea37c02c41a5bc7 from peer 248493ec312e4398b828c6cb00e5c4b1 at {username='slave'} at 127.0.0.1:32888: session id = 248493ec312e4398b828c6cb00e5c4b1-741e08995c924c8e9ea37c02c41a5bc7
I20251026 14:58:47.605350 22669 tablet_copy_source_session.cc:215] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 14:58:47.610149 23532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 741e08995c924c8e9ea37c02c41a5bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:47.621702 23536 ts_tablet_manager.cc:933] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1: Initiating tablet copy from peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:47.630189 23536 tablet_copy_client.cc:323] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.2.3:39645
I20251026 14:58:47.631784 22818 tablet_copy_service.cc:140] P 0d684445287b450a8f738b6ad4b59821: Received BeginTabletCopySession request for tablet 02799da19b36403499a63101b8418ffe from peer 248493ec312e4398b828c6cb00e5c4b1 ({username='slave'} at 127.0.0.1:49652)
I20251026 14:58:47.632233 22818 tablet_copy_service.cc:161] P 0d684445287b450a8f738b6ad4b59821: Beginning new tablet copy session on tablet 02799da19b36403499a63101b8418ffe from peer 248493ec312e4398b828c6cb00e5c4b1 at {username='slave'} at 127.0.0.1:49652: session id = 248493ec312e4398b828c6cb00e5c4b1-02799da19b36403499a63101b8418ffe
I20251026 14:58:47.638609 22818 tablet_copy_source_session.cc:215] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 14:58:47.641564 23536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02799da19b36403499a63101b8418ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:47.646296 23532 tablet_copy_client.cc:806] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Starting download of 0 data blocks...
I20251026 14:58:47.647001 23532 tablet_copy_client.cc:670] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Starting download of 1 WAL segments...
I20251026 14:58:47.651595 23532 tablet_copy_client.cc:538] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 14:58:47.657500 23536 tablet_copy_client.cc:806] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Starting download of 0 data blocks...
I20251026 14:58:47.658231 23536 tablet_copy_client.cc:670] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Starting download of 1 WAL segments...
I20251026 14:58:47.662360 23536 tablet_copy_client.cc:538] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 14:58:47.663817 23532 tablet_bootstrap.cc:492] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap starting.
I20251026 14:58:47.674274 23536 tablet_bootstrap.cc:492] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap starting.
I20251026 14:58:47.676991 23538 ts_tablet_manager.cc:933] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1: Initiating tablet copy from peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:58:47.706065 23532 tablet_bootstrap.cc:492] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 14:58:47.707520 23538 tablet_copy_client.cc:323] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.2.1:46185
I20251026 14:58:47.709430 23542 ts_tablet_manager.cc:933] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1: Initiating tablet copy from peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:47.709491 22669 tablet_copy_service.cc:140] P df3a0d1619ee48e8ab68ee0543a72a29: Received BeginTabletCopySession request for tablet a35eb71706aa4f97bf6c64430916aef5 from peer 248493ec312e4398b828c6cb00e5c4b1 ({username='slave'} at 127.0.0.1:32888)
I20251026 14:58:47.710104 22669 tablet_copy_service.cc:161] P df3a0d1619ee48e8ab68ee0543a72a29: Beginning new tablet copy session on tablet a35eb71706aa4f97bf6c64430916aef5 from peer 248493ec312e4398b828c6cb00e5c4b1 at {username='slave'} at 127.0.0.1:32888: session id = 248493ec312e4398b828c6cb00e5c4b1-a35eb71706aa4f97bf6c64430916aef5
I20251026 14:58:47.710753 23532 tablet_bootstrap.cc:492] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap complete.
I20251026 14:58:47.711661 23532 ts_tablet_manager.cc:1403] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1: Time spent bootstrapping tablet: real 0.048s user 0.019s sys 0.004s
I20251026 14:58:47.716518 23532 raft_consensus.cc:359] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:47.721474 23532 raft_consensus.cc:740] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Initialized, Role: LEARNER
I20251026 14:58:47.722330 23536 tablet_bootstrap.cc:492] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 14:58:47.722182 23532 consensus_queue.cc:260] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:47.723408 23536 tablet_bootstrap.cc:492] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap complete.
I20251026 14:58:47.724323 22669 tablet_copy_source_session.cc:215] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 14:58:47.725836 23536 ts_tablet_manager.cc:1403] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1: Time spent bootstrapping tablet: real 0.052s user 0.033s sys 0.003s
I20251026 14:58:47.728940 23536 raft_consensus.cc:359] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:47.729869 23536 raft_consensus.cc:740] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Initialized, Role: LEARNER
I20251026 14:58:47.731634 23536 consensus_queue.cc:260] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:47.734663 23538 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a35eb71706aa4f97bf6c64430916aef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:47.738559 23532 ts_tablet_manager.cc:1434] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1: Time spent starting tablet: real 0.027s user 0.011s sys 0.004s
I20251026 14:58:47.740326 22668 tablet_copy_service.cc:342] P df3a0d1619ee48e8ab68ee0543a72a29: Request end of tablet copy session 248493ec312e4398b828c6cb00e5c4b1-741e08995c924c8e9ea37c02c41a5bc7 received from {username='slave'} at 127.0.0.1:32888
I20251026 14:58:47.740772 22668 tablet_copy_service.cc:434] P df3a0d1619ee48e8ab68ee0543a72a29: ending tablet copy session 248493ec312e4398b828c6cb00e5c4b1-741e08995c924c8e9ea37c02c41a5bc7 on tablet 741e08995c924c8e9ea37c02c41a5bc7 with peer 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:58:47.741161 23536 ts_tablet_manager.cc:1434] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1: Time spent starting tablet: real 0.015s user 0.009s sys 0.000s
I20251026 14:58:47.742925 22818 tablet_copy_service.cc:342] P 0d684445287b450a8f738b6ad4b59821: Request end of tablet copy session 248493ec312e4398b828c6cb00e5c4b1-02799da19b36403499a63101b8418ffe received from {username='slave'} at 127.0.0.1:49652
I20251026 14:58:47.743386 22818 tablet_copy_service.cc:434] P 0d684445287b450a8f738b6ad4b59821: ending tablet copy session 248493ec312e4398b828c6cb00e5c4b1-02799da19b36403499a63101b8418ffe on tablet 02799da19b36403499a63101b8418ffe with peer 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:58:47.749752 23542 tablet_copy_client.cc:323] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.2.3:39645
I20251026 14:58:47.751618 22818 tablet_copy_service.cc:140] P 0d684445287b450a8f738b6ad4b59821: Received BeginTabletCopySession request for tablet ea1e7da57b65433b9e01cad8c4adeb75 from peer 248493ec312e4398b828c6cb00e5c4b1 ({username='slave'} at 127.0.0.1:49652)
I20251026 14:58:47.752170 22818 tablet_copy_service.cc:161] P 0d684445287b450a8f738b6ad4b59821: Beginning new tablet copy session on tablet ea1e7da57b65433b9e01cad8c4adeb75 from peer 248493ec312e4398b828c6cb00e5c4b1 at {username='slave'} at 127.0.0.1:49652: session id = 248493ec312e4398b828c6cb00e5c4b1-ea1e7da57b65433b9e01cad8c4adeb75
I20251026 14:58:47.762537 22818 tablet_copy_source_session.cc:215] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 14:58:47.763803 23538 tablet_copy_client.cc:806] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Starting download of 0 data blocks...
I20251026 14:58:47.764472 23538 tablet_copy_client.cc:670] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Starting download of 1 WAL segments...
I20251026 14:58:47.766474 23542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea1e7da57b65433b9e01cad8c4adeb75. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:47.776460 23538 tablet_copy_client.cc:538] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 14:58:47.782724 23542 tablet_copy_client.cc:806] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Starting download of 0 data blocks...
I20251026 14:58:47.783433 23542 tablet_copy_client.cc:670] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Starting download of 1 WAL segments...
I20251026 14:58:47.787138 23542 tablet_copy_client.cc:538] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 14:58:47.787493 23538 tablet_bootstrap.cc:492] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap starting.
I20251026 14:58:47.795753 23542 tablet_bootstrap.cc:492] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap starting.
I20251026 14:58:47.818553 23538 tablet_bootstrap.cc:492] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 14:58:47.819582 23538 tablet_bootstrap.cc:492] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap complete.
I20251026 14:58:47.820326 23538 ts_tablet_manager.cc:1403] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1: Time spent bootstrapping tablet: real 0.033s user 0.020s sys 0.005s
I20251026 14:58:47.824076 23538 raft_consensus.cc:359] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:47.826802 23538 raft_consensus.cc:740] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Initialized, Role: LEARNER
I20251026 14:58:47.827565 23538 consensus_queue.cc:260] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:47.830708 23538 ts_tablet_manager.cc:1434] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20251026 14:58:47.832991 22669 tablet_copy_service.cc:342] P df3a0d1619ee48e8ab68ee0543a72a29: Request end of tablet copy session 248493ec312e4398b828c6cb00e5c4b1-a35eb71706aa4f97bf6c64430916aef5 received from {username='slave'} at 127.0.0.1:32888
I20251026 14:58:47.833453 22669 tablet_copy_service.cc:434] P df3a0d1619ee48e8ab68ee0543a72a29: ending tablet copy session 248493ec312e4398b828c6cb00e5c4b1-a35eb71706aa4f97bf6c64430916aef5 on tablet a35eb71706aa4f97bf6c64430916aef5 with peer 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:58:47.834583 23542 tablet_bootstrap.cc:492] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 14:58:47.835856 23542 tablet_bootstrap.cc:492] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap complete.
I20251026 14:58:47.836573 23542 ts_tablet_manager.cc:1403] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1: Time spent bootstrapping tablet: real 0.041s user 0.023s sys 0.012s
I20251026 14:58:47.839795 23542 raft_consensus.cc:359] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:47.840683 23542 raft_consensus.cc:740] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Initialized, Role: LEARNER
I20251026 14:58:47.841370 23542 consensus_queue.cc:260] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:47.844199 23542 ts_tablet_manager.cc:1434] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20251026 14:58:47.845819 22818 tablet_copy_service.cc:342] P 0d684445287b450a8f738b6ad4b59821: Request end of tablet copy session 248493ec312e4398b828c6cb00e5c4b1-ea1e7da57b65433b9e01cad8c4adeb75 received from {username='slave'} at 127.0.0.1:49652
I20251026 14:58:47.846242 22818 tablet_copy_service.cc:434] P 0d684445287b450a8f738b6ad4b59821: ending tablet copy session 248493ec312e4398b828c6cb00e5c4b1-ea1e7da57b65433b9e01cad8c4adeb75 on tablet ea1e7da57b65433b9e01cad8c4adeb75 with peer 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:58:47.948408 23404 raft_consensus.cc:1217] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251026 14:58:48.086901 23404 raft_consensus.cc:1217] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251026 14:58:48.125526 23404 raft_consensus.cc:1217] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251026 14:58:48.225932 23404 raft_consensus.cc:1217] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251026 14:58:48.484548 23489 raft_consensus.cc:1064] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821: attempting to promote NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 to VOTER
I20251026 14:58:48.487795 23489 consensus_queue.cc:237] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:48.494853 23404 raft_consensus.cc:1275] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:48.495487 22729 raft_consensus.cc:1275] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:48.496493 22651 raft_consensus.cc:1275] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:48.496867 23446 consensus_queue.cc:1048] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:48.498605 23446 consensus_queue.cc:1048] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:48.505427 23528 raft_consensus.cc:1064] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29: attempting to promote NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 to VOTER
I20251026 14:58:48.506134 23549 consensus_queue.cc:1048] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:48.508244 23528 consensus_queue.cc:237] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:48.510744 23436 raft_consensus.cc:2955] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.528029 23404 raft_consensus.cc:2955] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.530493 23403 raft_consensus.cc:1275] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:48.532445 23550 consensus_queue.cc:1048] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:48.541497 22555 catalog_manager.cc:5649] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 14:58:48.544580 22725 raft_consensus.cc:2955] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.538276 22729 raft_consensus.cc:1275] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:48.538061 22651 raft_consensus.cc:2955] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.566237 22805 raft_consensus.cc:1275] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:48.566470 23512 consensus_queue.cc:1048] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:48.569659 23528 consensus_queue.cc:1048] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:48.583289 23512 raft_consensus.cc:2955] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.590315 22725 raft_consensus.cc:2955] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.599387 23404 raft_consensus.cc:2955] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.600526 22808 raft_consensus.cc:2955] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.631944 22552 catalog_manager.cc:5649] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.647686 22651 consensus_queue.cc:237] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:48.656005 23404 raft_consensus.cc:1275] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 14:58:48.656836 22805 raft_consensus.cc:1275] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 14:58:48.657676 23512 consensus_queue.cc:1048] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:58:48.663188 23528 consensus_queue.cc:1048] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:58:48.669139 23512 raft_consensus.cc:2955] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.672560 22805 consensus_queue.cc:237] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:48.672971 22800 raft_consensus.cc:2955] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.679783 22543 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 741e08995c924c8e9ea37c02c41a5bc7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 14:58:48.684134 23404 raft_consensus.cc:1275] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 14:58:48.686313 23404 raft_consensus.cc:2955] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.687781 23446 consensus_queue.cc:1048] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:58:48.693264 22553 catalog_manager.cc:5649] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: config changed from index 3 to 4, VOTER 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) evicted. New cstate: current_term: 1 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 14:58:48.698380 23549 raft_consensus.cc:2955] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.701372 22725 raft_consensus.cc:1275] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 14:58:48.720124 23404 raft_consensus.cc:2955] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.721789 23565 consensus_queue.cc:1048] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:58:48.724413 22540 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet ea1e7da57b65433b9e01cad8c4adeb75 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 14:58:48.726537 22725 raft_consensus.cc:2955] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.743989 22554 catalog_manager.cc:5649] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: config changed from index 3 to 4, VOTER df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) evicted. New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.755018 22640 tablet_service.cc:1552] Processing DeleteTablet for tablet ea1e7da57b65433b9e01cad8c4adeb75 with delete_type TABLET_DATA_TOMBSTONED (TS df3a0d1619ee48e8ab68ee0543a72a29 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54560
I20251026 14:58:48.758409 23566 tablet_replica.cc:333] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29: stopping tablet replica
I20251026 14:58:48.759509 23566 raft_consensus.cc:2243] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:48.760264 23566 raft_consensus.cc:2272] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:48.761510 23549 raft_consensus.cc:1064] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821: attempting to promote NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 to VOTER
I20251026 14:58:48.764079 23566 ts_tablet_manager.cc:1916] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 14:58:48.764544 23549 consensus_queue.cc:237] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:48.767382 22720 tablet_service.cc:1552] Processing DeleteTablet for tablet 741e08995c924c8e9ea37c02c41a5bc7 with delete_type TABLET_DATA_TOMBSTONED (TS 562bb1e1c8e445b8baecd29c3669658e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35078
I20251026 14:58:48.782078 23567 tablet_replica.cc:333] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e: stopping tablet replica
I20251026 14:58:48.783015 23567 raft_consensus.cc:2243] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:48.783893 23567 raft_consensus.cc:2272] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:48.786655 23404 raft_consensus.cc:1275] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:48.787647 22725 raft_consensus.cc:1275] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:48.788712 22651 raft_consensus.cc:1275] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:48.789531 23549 consensus_queue.cc:1048] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:48.790853 23564 consensus_queue.cc:1048] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:58:48.792768 23567 ts_tablet_manager.cc:1916] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 14:58:48.795097 23549 consensus_queue.cc:1048] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:48.812148 23566 ts_tablet_manager.cc:1929] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251026 14:58:48.812929 23566 log.cc:1199] T ea1e7da57b65433b9e01cad8c4adeb75 P df3a0d1619ee48e8ab68ee0543a72a29: Deleting WAL directory at /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-0-root/wals/ea1e7da57b65433b9e01cad8c4adeb75
I20251026 14:58:48.814968 22543 catalog_manager.cc:4985] TS df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): tablet ea1e7da57b65433b9e01cad8c4adeb75 (table test-workload [id=0253264dd7b746d58adac3c9eeddb55d]) successfully deleted
I20251026 14:58:48.824397 23567 ts_tablet_manager.cc:1929] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251026 14:58:48.825766 23567 log.cc:1199] T 741e08995c924c8e9ea37c02c41a5bc7 P 562bb1e1c8e445b8baecd29c3669658e: Deleting WAL directory at /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-1-root/wals/741e08995c924c8e9ea37c02c41a5bc7
I20251026 14:58:48.827540 22542 catalog_manager.cc:4985] TS 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): tablet 741e08995c924c8e9ea37c02c41a5bc7 (table test-workload [id=0253264dd7b746d58adac3c9eeddb55d]) successfully deleted
I20251026 14:58:48.827973 23489 raft_consensus.cc:2955] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.829814 23404 raft_consensus.cc:2955] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.836274 22651 raft_consensus.cc:2955] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.841141 23528 raft_consensus.cc:1064] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29: attempting to promote NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 to VOTER
I20251026 14:58:48.843716 23528 consensus_queue.cc:237] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:48.844270 22553 catalog_manager.cc:5649] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.856472 22725 raft_consensus.cc:2955] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } attrs { replace: true } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.862075 22805 raft_consensus.cc:1275] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:48.861898 23404 raft_consensus.cc:1275] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:48.868345 23563 consensus_queue.cc:1048] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:58:48.870143 22729 raft_consensus.cc:1275] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 14:58:48.870285 23563 consensus_queue.cc:1048] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:48.875857 22805 consensus_queue.cc:237] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:48.879598 23563 consensus_queue.cc:1048] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:58:48.883298 23403 raft_consensus.cc:1275] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 14:58:48.884775 23489 consensus_queue.cc:1048] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:58:48.886567 22725 raft_consensus.cc:1275] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 14:58:48.891764 23436 consensus_queue.cc:1048] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:58:48.893251 23436 raft_consensus.cc:2955] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.897568 22729 raft_consensus.cc:2955] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.901777 23403 raft_consensus.cc:2955] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.920459 22540 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 02799da19b36403499a63101b8418ffe with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 14:58:48.946807 23528 raft_consensus.cc:2955] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.955849 22553 catalog_manager.cc:5649] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: config changed from index 3 to 4, VOTER df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) evicted. New cstate: current_term: 1 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.970332 22640 tablet_service.cc:1552] Processing DeleteTablet for tablet 02799da19b36403499a63101b8418ffe with delete_type TABLET_DATA_TOMBSTONED (TS df3a0d1619ee48e8ab68ee0543a72a29 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54560
I20251026 14:58:48.971868 23566 tablet_replica.cc:333] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29: stopping tablet replica
I20251026 14:58:48.977830 23404 raft_consensus.cc:2955] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.986641 22805 raft_consensus.cc:2955] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:48.995834 22725 raft_consensus.cc:2955] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:49.008502 22552 catalog_manager.cc:5649] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: config changed from index 2 to 3, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:49.034181 23566 raft_consensus.cc:2243] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:49.035053 23566 raft_consensus.cc:2272] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:49.038105 22654 consensus_queue.cc:237] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:49.039593 23566 ts_tablet_manager.cc:1916] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 14:58:49.045622 22805 raft_consensus.cc:1275] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 14:58:49.046335 23404 raft_consensus.cc:1275] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 14:58:49.047250 23563 consensus_queue.cc:1048] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:58:49.048341 23528 consensus_queue.cc:1048] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:58:49.055895 23566 ts_tablet_manager.cc:1929] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251026 14:58:49.056473 23566 log.cc:1199] T 02799da19b36403499a63101b8418ffe P df3a0d1619ee48e8ab68ee0543a72a29: Deleting WAL directory at /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-0-root/wals/02799da19b36403499a63101b8418ffe
I20251026 14:58:49.056541 23563 raft_consensus.cc:2955] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:49.058259 22543 catalog_manager.cc:4985] TS df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): tablet 02799da19b36403499a63101b8418ffe (table test-workload [id=0253264dd7b746d58adac3c9eeddb55d]) successfully deleted
I20251026 14:58:49.058313 22800 raft_consensus.cc:2955] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:49.064821 23404 raft_consensus.cc:2955] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:49.067484 22543 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet a35eb71706aa4f97bf6c64430916aef5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 14:58:49.080752 22555 catalog_manager.cc:5649] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: config changed from index 3 to 4, VOTER 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) evicted. New cstate: current_term: 1 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:49.092720 22720 tablet_service.cc:1552] Processing DeleteTablet for tablet a35eb71706aa4f97bf6c64430916aef5 with delete_type TABLET_DATA_TOMBSTONED (TS 562bb1e1c8e445b8baecd29c3669658e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35078
I20251026 14:58:49.093860 23567 tablet_replica.cc:333] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e: stopping tablet replica
I20251026 14:58:49.094937 23567 raft_consensus.cc:2243] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 14:58:49.095732 23567 raft_consensus.cc:2272] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 14:58:49.099041 23567 ts_tablet_manager.cc:1916] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 14:58:49.118587 23567 ts_tablet_manager.cc:1929] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251026 14:58:49.118898 23567 log.cc:1199] T a35eb71706aa4f97bf6c64430916aef5 P 562bb1e1c8e445b8baecd29c3669658e: Deleting WAL directory at /tmp/dist-test-task_32k94/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1761490649341556-21512-0/minicluster-data/ts-1-root/wals/a35eb71706aa4f97bf6c64430916aef5
I20251026 14:58:49.120095 22542 catalog_manager.cc:4985] TS 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): tablet a35eb71706aa4f97bf6c64430916aef5 (table test-workload [id=0253264dd7b746d58adac3c9eeddb55d]) successfully deleted
I20251026 14:58:50.722903 23585 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:50.724740 23585 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:50.916437 23589 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:50.917455 23589 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:50.917819 23585 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:50.918754 23585 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:50.919947 23589 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:50.919754 23585 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:50.944192 23587 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:50.946197 23588 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:50.947526 23588 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:50.955711 21517 debug-util.cc:398] Leaking SignalData structure 0x7b080000f8e0 after lost signal to thread 22697
W20251026 14:58:50.956655 21517 kernel_stack_watchdog.cc:198] Thread 22697 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 280ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
I20251026 14:58:50.957544 23589 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:50.961661 23589 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.082656 23588 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.083794 23588 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:50.949645 23587 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.097203 23589 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:51.103137 23590 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.104081 23590 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.104818 23587 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.105697 23588 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.107668 23590 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.109802 23591 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.109957 23590 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.110894 23591 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:51.101222 23589 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.111774 23587 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.112357 23589 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.112510 23587 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.111816 23528 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.114079 23591 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:51.114722 23587 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.115032 23591 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.115381 23587 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.116040 23591 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.116232 23587 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.117645 23592 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.114760 23528 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.118549 23592 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.111027 23590 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.121415 23588 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.120098 23589 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:51.130352 23589 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:51.134522 23586 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.134177 23589 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.143414 23589 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:51.144440 23589 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:51.135288 23586 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.125205 23587 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.119091 23591 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.169478 23589 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.175885 23528 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.177093 23585 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:51.177552 23589 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:51.178370 23591 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.155972 23588 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.183176 23528 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.185751 23528 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:51.188261 22698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.550s user 0.027s sys 0.013s
I20251026 14:58:51.188585 23528 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:58:51.188870 22698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.550s user 0.027s sys 0.013s
I20251026 14:58:51.179370 23586 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.192305 23590 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.193230 23593 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.166666 23592 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.184481 23588 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.203291 23594 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.169229 23587 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.206892 23587 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.201723 23588 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.179392 23591 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.210125 23592 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.211128 23592 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.211805 23591 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.203263 23593 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.222074 23593 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:51.225471 23590 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.233500 23590 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.218526 23587 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.360813 23586 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.361831 23586 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:51.364555 23586 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:51.366720 23592 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.370692 23590 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.370981 23592 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.371287 23590 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.371721 23592 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.371953 23590 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.378262 23603 raft_consensus.cc:493] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:58:51.380985 22697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.710s user 0.003s sys 0.024s
W20251026 14:58:51.393713 22680 debug-util.cc:398] Leaking SignalData structure 0x7b0800149c40 after lost signal to thread 21515
W20251026 14:58:51.406744 22680 debug-util.cc:398] Leaking SignalData structure 0x7b08000b5060 after lost signal to thread 22590
I20251026 14:58:51.370298 23595 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.394567 23590 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.204414 23594 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.399930 23528 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:58:51.415678 22697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.744s user 0.004s sys 0.024s
W20251026 14:58:51.408564 22680 debug-util.cc:398] Leaking SignalData structure 0x7b080027cca0 after lost signal to thread 22683
I20251026 14:58:51.223472 23588 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.365223 23586 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.418293 23563 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.427053 23528 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.427222 23563 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.427520 23586 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:51.427901 23588 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.429628 23586 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:51.419785 23590 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.430298 23586 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.430892 23590 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.431586 23592 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.431456 23586 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.432251 23592 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.432999 23592 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.430274 23588 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.409900 23587 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.439273 23563 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.433231 23590 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.434593 23586 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.467634 23528 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.441834 23588 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.464951 23592 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.436633 23603 raft_consensus.cc:515] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:51.554153 23596 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.560034 23588 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.555254 23596 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.593550 23606 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.553493 23587 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.591706 23528 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.615669 23436 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.550333 23595 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.571806 23592 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.598984 23599 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:51.591297 23610 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
W20251026 14:58:51.618645 22699 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.943s user 0.012s sys 0.025s
I20251026 14:58:51.612572 23489 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:51.616748 23436 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.595898 23586 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.610639 23590 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.597514 23601 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.594377 23606 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.626554 23528 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.645071 23605 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.645911 23605 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:58:51.648151 22699 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.974s user 0.014s sys 0.025s
I20251026 14:58:51.652034 23592 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.653014 23604 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.653791 23604 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.645087 23489 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:51.683429 23609 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.696225 23436 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:51.679960 23603 leader_election.cc:290] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.674957 23590 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.677515 23601 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.677261 23586 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.725972 23489 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:51.691753 23592 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.750885 23588 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.828487 23608 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:51.686414 23610 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.858868 23606 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:51.829589 23613 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.863118 23593 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.861202 23436 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:51.866873 23607 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.866629 23593 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.851274 23603 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:51.851563 23592 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.863337 23489 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.856442 23605 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.871851 23436 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.872834 23618 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:51.849879 23563 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.856343 23596 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:51.874083 23528 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.877172 23604 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.878643 23598 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.878835 23436 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:51.686296 23599 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:51.879379 23603 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:51.867666 23607 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.877919 23605 raft_consensus.cc:493] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.884534 23605 raft_consensus.cc:515] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:51.885672 23603 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.885353 23592 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.886819 23605 leader_election.cc:290] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.882966 23563 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:58:51.890393 23563 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:58:51.890928 22700 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.226s user 0.035s sys 0.022s
I20251026 14:58:51.876807 23595 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.893612 23489 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:58:51.887719 23592 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.901460 23592 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.902711 23581 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.903321 23581 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.873493 23613 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.879302 23598 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.924582 23597 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.910358 23591 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.925361 23597 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.902267 23592 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.932080 23612 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.933841 23436 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:51.882988 23619 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.843307 23609 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.932861 23612 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.934674 23436 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.937224 23582 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:51.926049 23585 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.926252 23606 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.935004 23589 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:51.869943 23587 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.912312 23605 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.946152 23617 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.954087 23436 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:51.954440 23610 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:51.955044 23606 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.957422 23585 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.959537 23601 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.959721 23585 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:51.944401 23615 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:51.962343 23489 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:58:51.953853 23617 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:51.969532 23598 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:51.969848 23617 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.970360 23589 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.972414 23589 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:51.948118 23593 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:51.976197 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:51.964188 22680 debug-util.cc:398] Leaking SignalData structure 0x7b0800257860 after lost signal to thread 22758
I20251026 14:58:51.977423 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:51.975309 23613 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:51.970319 23581 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:51.977423 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:51.960753 23606 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:51.980674 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:51.978526 22652 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:51.985428 22654 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:51.931033 23618 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.973126 23590 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:51.990098 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:51.991110 22656 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:51.992609 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.000890 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:51.961730 23591 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.987469 23611 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:51.980496 23615 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.917941 23608 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.007925 23611 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:51.962157 23619 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:51.980144 23605 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:52.005364 23597 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:58:52.013777 22680 debug-util.cc:398] Leaking SignalData structure 0x7b08002eacc0 after lost signal to thread 22832
I20251026 14:58:52.008445 23586 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.016420 22650 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:52.004669 23582 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.050009 23608 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:58:52.052410 22700 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.388s user 0.036s sys 0.023s
W20251026 14:58:52.063078 22680 debug-util.cc:398] Leaking SignalData structure 0x7b0800295d80 after lost signal to thread 23428
I20251026 14:58:52.046628 23594 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.065095 23592 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.088361 23615 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:52.089480 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.090178 22653 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:52.091737 23605 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:52.100394 23591 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.094496 23612 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.110810 23619 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:52.113121 23618 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:52.114152 23609 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.114691 23620 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:52.095541 23599 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:52.119549 23616 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:52.120267 23616 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.120777 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.122356 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.122951 22653 raft_consensus.cc:2468] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 1.
I20251026 14:58:52.123957 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.124164 23370 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.125768 23607 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:52.125890 23605 raft_consensus.cc:2804] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.126518 23605 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:52.124573 22657 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 1.
I20251026 14:58:52.126905 23605 raft_consensus.cc:3060] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.128188 23616 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.115299 23620 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.131646 23620 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.133531 23614 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:52.133809 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.108801 23611 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:52.137809 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.139117 23617 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:52.139475 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.139755 23617 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:52.139132 22800 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 1.
I20251026 14:58:52.140100 22659 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:52.141548 23582 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:52.134222 23614 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.141837 23617 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.143121 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.146535 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.148109 23612 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:52.149396 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.151357 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.157343 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.158049 22805 raft_consensus.cc:2468] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 1.
I20251026 14:58:52.144964 23370 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.164247 23614 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.170524 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.171207 22651 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:52.160687 23612 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.174185 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.176657 23612 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.177541 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.177922 23621 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:52.178395 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.179556 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.180374 22653 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:52.181630 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.182221 22659 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:52.184717 22657 raft_consensus.cc:2468] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:52.186143 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.186988 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.184885 23624 raft_consensus.cc:2804] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.188201 22726 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.188118 22729 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.188246 23624 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.189297 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.189622 23624 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.190049 22733 raft_consensus.cc:2468] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.178635 23621 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.191568 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.193665 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.194476 22728 raft_consensus.cc:2468] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.188328 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.193302 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.190138 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:58:52.181751 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 46422.3us
I20251026 14:58:52.198362 22732 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.192800 23612 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:52.198864 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.199481 22657 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:52.199349 23612 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.200106 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.207341 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.207842 23616 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:52.208016 22659 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:52.208434 23616 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.200992 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.211555 23625 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:52.212378 23625 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.223066 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:58:52.251262 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.252483 22725 raft_consensus.cc:2468] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 1.
I20251026 14:58:52.263469 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.263707 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:58:52.263808 23612 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.263882 23616 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.264340 23403 raft_consensus.cc:2468] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:52.264669 23616 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:52.265408 23611 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:52.265266 23616 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.265996 23611 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.267315 23616 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.267850 23616 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:52.267900 23611 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.268497 23611 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:52.270342 23614 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:52.270538 22730 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.272982 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.272491 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.275269 22653 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:52.268427 23616 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.275336 22659 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:52.270393 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.276134 23602 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:52.276499 23614 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.277753 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.276706 23602 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.278573 23616 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.278643 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.279330 22725 raft_consensus.cc:2468] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 1.
I20251026 14:58:52.279565 23614 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.281069 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.283324 23602 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.283424 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.284045 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.285365 23404 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:58:52.270396 23611 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.285943 23404 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:52.271589 23621 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.287696 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.288278 22801 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.289031 23611 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.289822 23404 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:58:52.290345 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.291200 22730 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.289803 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:58:52.290125 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:58:52.292037 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.290364 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.290940 22725 raft_consensus.cc:2468] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 1.
I20251026 14:58:52.293258 22657 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:52.293581 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.294132 22659 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:52.295526 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.295531 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.296941 23625 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.297467 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.297554 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.298259 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.288694 23624 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:52.303978 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.304412 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.304625 22650 raft_consensus.cc:2468] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:52.296511 22659 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:52.305589 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.299602 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.306170 22651 raft_consensus.cc:2468] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:52.300796 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.306329 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.298465 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.307052 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.307358 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.307673 22730 raft_consensus.cc:2468] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.303268 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.308025 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.308699 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.300055 23605 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:52.310166 23400 raft_consensus.cc:2393] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 248493ec312e4398b828c6cb00e5c4b1 in this term.
I20251026 14:58:52.310683 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.310853 23402 raft_consensus.cc:1240] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:52.311266 22658 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:52.311631 23605 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.312436 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.313402 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:52.313567 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.306172 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:52.314411 22650 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.314953 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.315198 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.309449 22806 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.317040 22771 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters:
I20251026 14:58:52.309686 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.318104 23622 raft_consensus.cc:2804] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.318482 23622 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:52.318835 23622 raft_consensus.cc:3060] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.319253 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.319907 22734 raft_consensus.cc:2468] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.319999 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.321177 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.320618 22725 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.322412 22733 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.322640 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.323311 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.323856 22732 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.324841 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.326020 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.328531 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.329360 22771 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters:
I20251026 14:58:52.330376 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.317015 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.330504 23582 raft_consensus.cc:2804] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.334014 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.334465 23582 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:52.334848 23582 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.336061 23625 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:52.337633 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.339195 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.340340 23625 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.342759 23625 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.344492 23582 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:52.345619 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.346508 23582 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:52.346686 23624 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:52.336771 23622 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:52.350630 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.351254 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.350438 23369 leader_election.cc:304] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:52.354504 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.355630 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.356724 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.355104 22650 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 2.
I20251026 14:58:52.358181 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.358649 22730 raft_consensus.cc:2468] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.360471 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.357084 23624 raft_consensus.cc:2804] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.361603 23624 raft_consensus.cc:493] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:52.361858 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.362573 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.362721 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.363147 23369 leader_election.cc:304] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:52.363462 22732 raft_consensus.cc:2468] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:52.363399 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.364135 23369 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:52.364527 22733 raft_consensus.cc:2468] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.364941 23623 raft_consensus.cc:2804] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.365378 23623 raft_consensus.cc:493] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:52.365449 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.365765 23623 raft_consensus.cc:3060] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.366212 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.367331 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.367429 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.369637 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.370201 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.370409 23370 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.370882 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.371320 23617 raft_consensus.cc:2804] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:52.371819 23617 raft_consensus.cc:697] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 LEADER]: Becoming Leader. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:58:52.372821 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.372808 23617 consensus_queue.cc:237] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:52.370266 22733 raft_consensus.cc:2468] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.375145 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.379954 22730 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.375674 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.366894 23622 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:52.366847 23624 raft_consensus.cc:3060] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.375847 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:52.383970 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.375742 23605 raft_consensus.cc:2804] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.375090 23582 consensus_queue.cc:1059] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20251026 14:58:52.387519 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.386721 23605 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:52.388983 23605 raft_consensus.cc:3060] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.389340 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:58:52.385978 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.390364 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:52.391433 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.392916 23582 raft_consensus.cc:3055] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:52.393230 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:52.393380 23582 raft_consensus.cc:740] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:52.394918 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.393790 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.397969 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.399518 22697 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:58:52.400439 23614 raft_consensus.cc:2749] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:52.403030 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.403656 22774 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.404536 23599 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:52.404386 23582 consensus_queue.cc:260] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:52.404924 23599 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.405738 23599 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:52.405845 23582 raft_consensus.cc:3060] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.413555 23599 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.414934 22774 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.416456 22774 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.418751 23607 raft_consensus.cc:2804] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.419130 23607 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.419526 23607 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.418198 22554 catalog_manager.cc:5649] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: term changed from 1 to 2, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4). New cstate: current_term: 2 leader_uuid: "248493ec312e4398b828c6cb00e5c4b1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 14:58:52.419842 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:58:52.420128 23618 raft_consensus.cc:2804] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.420845 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.420974 23618 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.378640 23623 raft_consensus.cc:515] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:52.419657 23599 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:52.421414 23618 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.423357 22774 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.426745 22774 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.427673 23615 raft_consensus.cc:2804] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.427729 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.428320 23615 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.428853 23615 raft_consensus.cc:3060] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.428977 23599 raft_consensus.cc:2804] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:52.429493 23599 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.430995 23599 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:52.437907 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.439204 23402 raft_consensus.cc:1240] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:52.444999 23623 leader_election.cc:290] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.447418 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.447983 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.450498 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.451643 23622 consensus_queue.cc:1059] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 14:58:52.452169 22774 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.455958 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.457442 23597 raft_consensus.cc:2804] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.457672 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.457909 23597 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.458240 22808 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.458407 23597 raft_consensus.cc:3060] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.460923 22774 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.465976 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.466571 22730 raft_consensus.cc:2468] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.467561 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:52.467957 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.470168 22725 raft_consensus.cc:3060] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.470628 22808 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.462182 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.472440 22774 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:58:52.521623 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 49257.2us
I20251026 14:58:52.522343 22774 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.467649 23619 raft_consensus.cc:2804] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.510607 22725 raft_consensus.cc:2468] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 2.
I20251026 14:58:52.523154 22774 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.486178 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.499578 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.504132 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.528030 22774 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.528896 22774 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.528092 23608 raft_consensus.cc:3055] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:52.537456 23608 raft_consensus.cc:740] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:52.523464 23581 raft_consensus.cc:2804] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.539527 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.540108 22725 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.523152 23624 raft_consensus.cc:515] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:52.533471 23622 raft_consensus.cc:2804] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.526312 23619 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.535346 23582 raft_consensus.cc:2804] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.535091 23606 raft_consensus.cc:2804] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.534236 23618 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.540460 23581 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.549935 23581 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.543143 23622 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.552421 23622 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.552647 23607 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.557751 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.557859 23597 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.545903 23608 consensus_queue.cc:260] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:52.544653 23606 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.559290 23606 raft_consensus.cc:3060] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.553395 23402 raft_consensus.cc:1240] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:52.556452 23618 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:52.543510 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.563319 23618 raft_consensus.cc:2804] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.565353 23367 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:52.543337 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.566246 23623 raft_consensus.cc:2804] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.566620 23623 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:52.566983 23623 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.568166 23367 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:52.569638 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.570575 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.571281 22806 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.563921 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.544245 23619 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.567878 23615 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.544301 23582 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.568918 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.570740 23617 raft_consensus.cc:2804] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.563983 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.568341 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.565512 23618 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.576093 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:52.562515 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.576498 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.580201 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.580487 22622 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.577323 23599 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.580751 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.592015 22725 raft_consensus.cc:2468] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.592463 23597 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:52.594116 23599 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:52.582451 23615 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:52.588577 22730 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
W20251026 14:58:52.589289 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 60415.2us
I20251026 14:58:52.588732 22732 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:52.542712 23369 leader_election.cc:304] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:52.598364 22774 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.575471 23608 raft_consensus.cc:3060] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.599319 23607 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:52.581202 23617 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:52.609674 23617 raft_consensus.cc:3060] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.609494 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.581066 23582 raft_consensus.cc:3060] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.614450 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.616187 23606 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.606040 22622 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:58:52.608125 22659 raft_consensus.cc:1240] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:52.589362 23618 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.604367 23609 raft_consensus.cc:2804] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.617610 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.628973 23609 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.629467 23609 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.617684 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.618527 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.635221 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:52.620222 23624 leader_election.cc:290] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.620819 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.624192 22622 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.637972 23591 raft_consensus.cc:2804] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.638381 23591 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.638725 23591 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.626485 22624 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:52.624794 23592 raft_consensus.cc:2749] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:52.627058 23603 raft_consensus.cc:2804] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:52.642141 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.628031 23606 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:58:52.625070 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:52.631345 22658 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:52.608685 23605 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:52.645318 23582 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.646611 23605 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.646835 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:52.577330 23623 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:52.643187 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.643544 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.647742 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.648168 22652 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:52.648363 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.648828 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:52.649818 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:52.650530 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 52281.3us
I20251026 14:58:52.652127 22774 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.652848 22624 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:52.653877 22624 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:52.655557 23597 raft_consensus.cc:2804] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.656018 23597 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.656420 23597 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.651759 23603 raft_consensus.cc:697] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 2 LEADER]: Becoming Leader. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:58:52.657713 23603 consensus_queue.cc:237] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:58:52.624852 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 82679.8us
I20251026 14:58:52.662637 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.664631 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:52.662906 23592 raft_consensus.cc:2804] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.671841 23592 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:52.672303 23592 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.644307 23634 raft_consensus.cc:2804] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.674852 23634 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:52.665908 23625 raft_consensus.cc:2804] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.665983 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:52.675707 23625 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:52.676153 23625 raft_consensus.cc:3060] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.676280 23402 raft_consensus.cc:1240] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 LEADER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:52.668881 22774 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.675390 23634 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.698932 22622 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.707823 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:58:52.732131 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 63300.2us
I20251026 14:58:52.754429 22734 raft_consensus.cc:3060] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.710071 23402 raft_consensus.cc:1240] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:52.696650 22727 raft_consensus.cc:1240] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
W20251026 14:58:52.748636 22773 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.573s (SENT) [suppressed 3 similar messages]
I20251026 14:58:52.692885 23615 consensus_queue.cc:1059] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20251026 14:58:52.696642 23602 consensus_queue.cc:1059] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20251026 14:58:52.651540 23582 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:52.754715 22774 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.761421 22808 raft_consensus.cc:2468] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 2.
I20251026 14:58:52.760901 23623 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:58:52.764089 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 110250us
I20251026 14:58:52.684612 23606 raft_consensus.cc:2804] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.765663 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.766876 22624 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:58:52.767381 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 68553.8us
I20251026 14:58:52.707419 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:52.770957 22733 raft_consensus.cc:3060] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.771530 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.769948 22622 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.774221 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.759209 23633 raft_consensus.cc:2804] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.664009 23581 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.775619 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.777192 22774 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.779366 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:52.768992 23606 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.779949 23633 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.830534 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.826236 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:58:52.831526 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 54435.5us
I20251026 14:58:52.781069 23602 consensus_queue.cc:1243] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer 248493ec312e4398b828c6cb00e5c4b1 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20251026 14:58:52.833671 22622 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.836758 23608 consensus_queue.cc:1059] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 14:58:52.840781 23625 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.834847 22774 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:52.823361 23582 raft_consensus.cc:2804] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.835214 23367 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:52.832737 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.848961 22734 raft_consensus.cc:2468] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 2.
I20251026 14:58:52.833248 23606 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.834908 23633 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.842111 23615 raft_consensus.cc:2804] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.856418 22733 raft_consensus.cc:2468] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 2.
I20251026 14:58:52.853806 22732 consensus_queue.cc:237] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:52.881121 23600 raft_consensus.cc:3055] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:52.883301 23600 raft_consensus.cc:740] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
W20251026 14:58:52.881238 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 51029.4us
I20251026 14:58:52.884220 23600 consensus_queue.cc:260] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:52.866345 23582 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
W20251026 14:58:52.887741 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 52965.7us
I20251026 14:58:52.895363 23642 raft_consensus.cc:2804] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.895996 23600 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.878852 23640 raft_consensus.cc:2804] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.881424 23603 raft_consensus.cc:2804] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:52.880442 23607 raft_consensus.cc:3055] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:52.848986 23619 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.877995 23615 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.853237 23634 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.890666 22774 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:58:52.885536 22771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 54078.1us
I20251026 14:58:52.894248 23582 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.881995 23638 raft_consensus.cc:2749] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:52.893510 23632 raft_consensus.cc:3055] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:52.903374 23607 raft_consensus.cc:740] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:52.897333 23642 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.907560 23615 raft_consensus.cc:3060] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.913164 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.899600 23640 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.897066 23608 raft_consensus.cc:2804] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.915398 23637 raft_consensus.cc:2804] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.915621 22733 raft_consensus.cc:1240] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:52.949852 23637 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.953699 23637 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
W20251026 14:58:52.956808 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 123089us
I20251026 14:58:52.915171 23597 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.959729 23597 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:52.959790 23619 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:52.911008 22774 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:58:52.931015 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 182491us
W20251026 14:58:52.961021 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 50074.9us
I20251026 14:58:52.949859 23608 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.908119 23603 raft_consensus.cc:697] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 2 LEADER]: Becoming Leader. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:58:52.909895 23581 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:52.910213 23632 raft_consensus.cc:740] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
W20251026 14:58:52.936192 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 169335us
I20251026 14:58:52.938216 22726 raft_consensus.cc:1240] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:52.911938 23641 raft_consensus.cc:2804] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.914829 23622 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.919739 23609 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.896878 23369 leader_election.cc:304] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:52.945535 23625 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.948453 23640 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.970966 22624 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:52.944968 23642 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.975898 23632 consensus_queue.cc:260] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:52.975594 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.975808 22624 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:52.938210 23617 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:52.977624 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.978221 22624 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:52.971576 23608 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.976771 23634 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:52.980880 22624 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:52.972287 23597 raft_consensus.cc:2804] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.972185 23641 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:52.988554 22774 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:52.980336 23638 raft_consensus.cc:2804] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.991818 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:52.969777 23607 consensus_queue.cc:260] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:52.993008 23607 raft_consensus.cc:3060] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.989233 23641 raft_consensus.cc:3060] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.989215 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.982846 23623 raft_consensus.cc:2804] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:52.990871 23638 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:52.981640 23625 raft_consensus.cc:2749] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:52.983140 23632 raft_consensus.cc:3060] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.991520 22774 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:52.979373 23644 raft_consensus.cc:2804] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:52.994223 23619 raft_consensus.cc:2749] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:58:52.975203 23618 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.994961 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:52.998513 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:52.986725 23597 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:53.007455 23597 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:52.978677 23603 consensus_queue.cc:237] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:52.995738 23609 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:53.000972 23644 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:52.997452 23638 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.006816 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:58:52.988265 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 102852us
I20251026 14:58:53.027423 23618 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:58:53.028697 22697 consensus_peers.cc:597] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e -> Peer 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341): Couldn't send request to peer 248493ec312e4398b828c6cb00e5c4b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2efb26bccebf43b5b8a85f51fd977714. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:53.029170 23617 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:58:53.070636 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:58:53.112985 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 84522.6us
I20251026 14:58:53.074141 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.114822 23606 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.116235 22807 raft_consensus.cc:1240] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.030232 23644 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.077539 22803 raft_consensus.cc:1240] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.116969 23606 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:52.998860 23646 raft_consensus.cc:2749] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:53.079857 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:53.083853 22800 raft_consensus.cc:1240] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.013409 23619 raft_consensus.cc:2749] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:58:53.091734 22808 raft_consensus.cc:1240] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 2. Current term is 3. Ops: []
I20251026 14:58:53.097966 22802 raft_consensus.cc:1240] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.098932 22804 raft_consensus.cc:1240] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
W20251026 14:58:53.121428 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 140620us
I20251026 14:58:53.107582 23622 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:53.108238 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.109004 23608 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.128116 22803 raft_consensus.cc:1240] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.109957 23597 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:52.997697 23623 raft_consensus.cc:697] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 LEADER]: Becoming Leader. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:58:53.125542 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.126461 22804 raft_consensus.cc:1240] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.125576 22808 raft_consensus.cc:1240] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.129870 22800 raft_consensus.cc:1240] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.129257 23608 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:53.129321 22802 raft_consensus.cc:1240] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.135773 22624 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
W20251026 14:58:53.136199 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:53.136828 22804 raft_consensus.cc:1240] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.137351 22624 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:53.143004 22804 raft_consensus.cc:1240] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
W20251026 14:58:53.144465 23591 consensus_meta.cc:287] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.505s user 0.013s sys 0.007s
I20251026 14:58:53.144608 22799 raft_consensus.cc:2468] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 2.
I20251026 14:58:53.145980 23616 consensus_queue.cc:1059] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:58:53.146202 23592 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.147428 23616 consensus_queue.cc:1059] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:53.148058 23602 raft_consensus.cc:3055] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:58:53.148311 23592 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:53.148443 23602 raft_consensus.cc:740] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:53.148624 23611 consensus_queue.cc:1059] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:58:53.149078 23647 raft_consensus.cc:2804] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:53.149729 22624 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:53.151245 22624 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:53.159739 22624 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:53.163720 22803 raft_consensus.cc:1240] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.164328 23609 raft_consensus.cc:2749] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:53.148371 23625 raft_consensus.cc:2749] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:53.149525 23647 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:53.165855 22624 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:53.144932 23591 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.167106 22624 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:53.174705 22624 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:53.176431 22624 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:53.161201 23652 consensus_queue.cc:1059] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20251026 14:58:53.166960 23592 raft_consensus.cc:2804] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:53.190016 22624 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:53.191736 22624 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:53.193163 22624 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:53.175755 23647 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.176199 23646 raft_consensus.cc:2804] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:53.175107 23625 raft_consensus.cc:2749] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:58:53.148471 23621 consensus_queue.cc:1059] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20251026 14:58:53.180495 23609 raft_consensus.cc:2804] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:53.180027 23634 raft_consensus.cc:2804] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:53.163473 23633 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.161854 23640 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:53.209304 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:53.195523 23592 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:53.197530 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.158046 23614 consensus_queue.cc:1059] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20251026 14:58:53.197378 23643 raft_consensus.cc:2804] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:53.149612 23602 consensus_queue.cc:260] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.218122 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.201298 23646 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:58:53.222047 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 60875.6us
I20251026 14:58:53.271694 22801 raft_consensus.cc:1240] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.200420 23651 raft_consensus.cc:2804] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:53.273312 23651 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:53.273715 23651 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.212880 23623 consensus_queue.cc:237] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:53.210219 23634 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:53.212399 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.205456 23609 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:58:53.215701 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:53.218236 23592 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.218132 23625 raft_consensus.cc:2749] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:53.290937 23402 raft_consensus.cc:1240] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.277966 23646 raft_consensus.cc:3060] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.203217 23369 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:53.275537 23616 raft_consensus.cc:3055] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:53.224381 23643 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:53.287629 23634 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.299979 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.292972 23609 raft_consensus.cc:3060] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
W20251026 14:58:53.303892 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 104560us
I20251026 14:58:53.304136 23616 raft_consensus.cc:740] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:53.312016 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.306010 23617 raft_consensus.cc:2804] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Leader election won for term 2
W20251026 14:58:53.301427 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:53.313795 23625 raft_consensus.cc:2749] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:53.309165 23643 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.316902 23617 raft_consensus.cc:697] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 LEADER]: Becoming Leader. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:58:53.310737 23602 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:53.296998 23637 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.313499 23597 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:53.317658 22803 raft_consensus.cc:1275] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20251026 14:58:53.331048 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 137983us
I20251026 14:58:53.336020 22624 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:53.330943 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:53.337881 22624 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:53.338943 22624 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:53.339681 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:53.337098 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:53.316980 23654 consensus_queue.cc:1059] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20251026 14:58:53.341218 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:53.341923 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:53.342545 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:53.355541 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:53.355592 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:53.351670 23656 raft_consensus.cc:2804] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:53.358197 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.342736 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.367247 23605 consensus_queue.cc:1048] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251026 14:58:53.367842 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.369096 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.369626 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:53.346853 23591 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:53.376343 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.376377 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:53.378571 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:53.359138 23656 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:53.386929 23652 raft_consensus.cc:3055] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:53.387490 23652 raft_consensus.cc:740] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:53.369096 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.392966 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:53.394162 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.377522 23633 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:53.403061 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:53.357930 23660 raft_consensus.cc:2804] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:53.379284 23640 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:53.385700 23656 raft_consensus.cc:3060] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.388701 23652 consensus_queue.cc:260] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.391502 23591 raft_consensus.cc:2804] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:53.413924 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:53.414779 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:53.389842 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.423940 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:53.390233 23654 consensus_queue.cc:1059] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20251026 14:58:53.400192 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:53.409101 22656 raft_consensus.cc:1240] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.382184 23611 raft_consensus.cc:3055] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:53.425715 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:53.372907 23616 consensus_queue.cc:260] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.426331 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.405524 23660 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:53.413815 23591 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:53.378847 23617 consensus_queue.cc:237] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:53.446127 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:53.445142 23591 raft_consensus.cc:3060] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.446398 23637 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:53.457901 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:53.424769 23614 raft_consensus.cc:3055] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:53.444746 23660 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:53.460985 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.422766 23655 raft_consensus.cc:3055] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:53.443712 23611 raft_consensus.cc:740] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:53.403971 23658 consensus_queue.cc:1059] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20251026 14:58:53.453655 23652 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.470094 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:53.454095 23623 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:53.476328 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:53.449157 23592 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.482486 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.477063 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:53.477128 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:53.477946 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.478229 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.478698 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:53.460135 23614 raft_consensus.cc:740] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
W20251026 14:58:53.472110 23582 consensus_meta.cc:287] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.546s user 0.011s sys 0.010s
I20251026 14:58:53.487214 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.465685 23655 raft_consensus.cc:740] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:53.473874 23616 raft_consensus.cc:3060] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.489202 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:53.491132 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:58:53.485697 23642 consensus_meta.cc:287] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.509s user 0.015s sys 0.006s
I20251026 14:58:53.459743 23646 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.497166 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:53.462785 23641 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.480593 23657 raft_consensus.cc:3055] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:53.508212 22808 raft_consensus.cc:1240] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.491781 23582 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:58:53.509774 23641 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:53.512662 23655 consensus_queue.cc:260] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:53.513242 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:53.519802 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:53.521148 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:53.521361 22657 raft_consensus.cc:1240] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.521778 22654 raft_consensus.cc:1240] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.522360 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:53.522804 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:53.523005 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:53.523756 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:53.484156 23623 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:58:53.504287 23615 consensus_meta.cc:287] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.553s user 0.005s sys 0.014s
I20251026 14:58:53.523836 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:53.547430 22654 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.543145 22650 raft_consensus.cc:1240] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.548985 23646 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:53.513952 23655 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.554514 22806 raft_consensus.cc:1240] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
W20251026 14:58:53.555114 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 216231us
I20251026 14:58:53.555896 22624 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:53.557050 22624 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:53.513952 23658 raft_consensus.cc:3055] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:53.558059 22624 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:53.510804 23657 raft_consensus.cc:740] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:53.540274 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:58:53.559612 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 280264us
I20251026 14:58:53.516141 23642 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.561439 22624 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:53.568159 22624 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:53.570724 22624 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:58:53.565186 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:53.560024 23646 raft_consensus.cc:2749] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:53.560635 23637 raft_consensus.cc:2749] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:58:53.560056 23658 raft_consensus.cc:740] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:53.561919 23640 raft_consensus.cc:2749] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:53.579818 23664 consensus_queue.cc:1059] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20251026 14:58:53.515882 23611 consensus_queue.cc:260] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.552568 23615 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.571405 23625 raft_consensus.cc:2804] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:53.573674 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:53.578706 23638 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.514277 23614 consensus_queue.cc:260] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:53.597890 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 118100us
I20251026 14:58:53.593654 22624 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:53.610869 23653 raft_consensus.cc:3055] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:53.611403 23653 raft_consensus.cc:740] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:53.588706 23592 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:53.605455 22805 raft_consensus.cc:1240] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.606105 22653 raft_consensus.cc:1240] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.615057 22624 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:58:53.616164 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:53.609118 22654 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:53.618048 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:53.589519 23625 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:53.595089 23646 raft_consensus.cc:2749] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:53.587476 23662 raft_consensus.cc:2804] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:53.581983 23644 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.621862 22657 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.623517 23640 raft_consensus.cc:2749] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:53.598738 23652 consensus_queue.cc:1059] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20251026 14:58:53.632581 23662 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:53.658308 23662 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.662494 23615 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:53.664011 23623 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:53.665635 23609 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.668573 23609 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:53.635116 23664 raft_consensus.cc:3055] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:53.669560 22650 raft_consensus.cc:1240] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.672451 22650 raft_consensus.cc:1240] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.631855 23625 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.612571 23653 consensus_queue.cc:260] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.674539 23653 raft_consensus.cc:3060] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.624812 23657 consensus_queue.cc:260] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.634776 23658 consensus_queue.cc:260] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.678756 23657 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.679440 23658 raft_consensus.cc:3060] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.629792 23611 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.681921 22656 raft_consensus.cc:1240] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.632227 23592 raft_consensus.cc:2749] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251026 14:58:53.686424 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 71497.4us
I20251026 14:58:53.625223 23621 consensus_queue.cc:1059] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
W20251026 14:58:53.688556 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 110417us
I20251026 14:58:53.688472 22624 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:53.690352 22624 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:53.691498 22624 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:58:53.661043 23600 consensus_meta.cc:287] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.760s user 0.005s sys 0.015s
W20251026 14:58:53.717691 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 77064.7us
I20251026 14:58:53.723636 22733 raft_consensus.cc:1275] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251026 14:58:53.669625 23664 raft_consensus.cc:740] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:53.672823 23643 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.682564 23638 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:53.727032 23669 raft_consensus.cc:2749] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251026 14:58:53.730034 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:53.730338 22554 catalog_manager.cc:5649] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 1 to 2, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4). New cstate: current_term: 2 leader_uuid: "248493ec312e4398b828c6cb00e5c4b1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:58:53.711934 23640 raft_consensus.cc:2749] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:53.748222 22653 raft_consensus.cc:1240] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.725209 23592 raft_consensus.cc:2804] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:53.720117 23582 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
W20251026 14:58:53.759327 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 69866.2us
I20251026 14:58:53.751951 23592 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:53.751952 23654 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:58:53.772784 23607 consensus_meta.cc:287] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.777s user 0.017s sys 0.012s
I20251026 14:58:53.750725 23642 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:53.768057 23673 raft_consensus.cc:3055] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:53.777801 23592 raft_consensus.cc:3060] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.794054 23644 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:53.798117 23643 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:53.786738 23673 raft_consensus.cc:740] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:53.799571 23634 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.800797 22654 raft_consensus.cc:1240] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.769735 23664 consensus_queue.cc:260] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:53.805084 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 164856us
I20251026 14:58:53.800149 23673 consensus_queue.cc:260] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:53.804576 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:53.793125 22657 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:53.810925 23656 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.784288 22728 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:53.819960 23666 consensus_queue.cc:1059] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:58:53.839074 23666 raft_consensus.cc:3055] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:53.839447 23666 raft_consensus.cc:740] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:53.841106 22658 raft_consensus.cc:1240] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
W20251026 14:58:53.797091 23647 consensus_meta.cc:287] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.594s user 0.015s sys 0.000s
I20251026 14:58:53.821679 23662 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.851570 23662 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:53.824466 22803 raft_consensus.cc:1275] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251026 14:58:53.826831 23661 consensus_queue.cc:1059] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:53.853034 23656 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:53.853725 23661 raft_consensus.cc:3055] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:53.853514 23591 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.854118 23661 raft_consensus.cc:740] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:53.779619 23667 consensus_queue.cc:1059] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
W20251026 14:58:53.796979 23651 consensus_meta.cc:287] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.523s user 0.010s sys 0.003s
I20251026 14:58:53.815125 23625 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.840271 23666 consensus_queue.cc:260] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.843997 22731 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:58:53.802168 23664 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.848609 22734 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:53.857218 22651 raft_consensus.cc:1240] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.858839 23634 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:53.788888 23654 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.836098 23614 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.845606 23602 consensus_queue.cc:1059] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20251026 14:58:53.860970 23667 raft_consensus.cc:3055] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:53.855880 23647 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:53.870539 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:58:53.866281 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 174840us
I20251026 14:58:53.875424 22624 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:53.866015 23617 consensus_queue.cc:1048] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20251026 14:58:53.871552 23673 raft_consensus.cc:3060] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.905999 22659 raft_consensus.cc:1240] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.874468 22650 raft_consensus.cc:1240] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.872221 23667 raft_consensus.cc:740] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:53.859673 23651 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.917533 23665 raft_consensus.cc:2804] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:53.926420 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:53.928229 22808 raft_consensus.cc:1240] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.924445 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:53.929111 22805 raft_consensus.cc:2393] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:53.927066 23592 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.927453 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:53.930723 22801 raft_consensus.cc:2393] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
W20251026 14:58:53.929553 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 127474us
I20251026 14:58:53.935644 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:53.940909 22806 raft_consensus.cc:2393] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:53.941115 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:53.945329 22651 raft_consensus.cc:1275] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251026 14:58:53.948347 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:53.948347 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:53.949764 22700 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:53.953773 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:53.936187 23666 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.957880 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:53.939493 23665 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:53.960075 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:53.952976 23608 consensus_queue.cc:1059] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20251026 14:58:53.858866 23661 consensus_queue.cc:260] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.965600 23658 raft_consensus.cc:3055] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:53.966284 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:53.967476 22657 raft_consensus.cc:3060] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.967134 23370 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:53.940296 23657 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:53.929494 23615 consensus_queue.cc:1059] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20251026 14:58:53.992956 22774 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:53.970945 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:53.995592 22700 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:53.947726 22655 raft_consensus.cc:1240] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:53.975157 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:54.002817 22808 raft_consensus.cc:2393] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:53.942514 23591 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:53.987901 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:53.949307 23654 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:53.991664 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:54.008183 23370 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:54.009455 23676 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:54.009678 23617 raft_consensus.cc:2749] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:58:53.999490 23647 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:53.961340 23670 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:53.973618 23665 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:53.990023 23592 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:53.968502 23625 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:53.973220 23607 consensus_queue.cc:1059] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20251026 14:58:53.967672 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:54.010314 23617 consensus_queue.cc:1048] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20251026 14:58:54.010051 23676 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.014066 23605 raft_consensus.cc:2749] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:58:54.014925 23667 consensus_queue.cc:260] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.019791 23600 raft_consensus.cc:3055] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:53.982610 23658 raft_consensus.cc:740] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:54.023912 23641 raft_consensus.cc:2804] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:54.024410 23670 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:53.978060 23621 raft_consensus.cc:2749] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:54.006103 23616 raft_consensus.cc:2749] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:54.033412 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:54.034029 22800 raft_consensus.cc:1275] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20251026 14:58:54.037736 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 252018us
I20251026 14:58:54.000617 23597 consensus_queue.cc:1059] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20251026 14:58:54.046362 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:54.048085 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:54.048185 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:54.049111 22804 raft_consensus.cc:1240] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:54.058403 22700 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:54.048264 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:58:54.031483 23660 consensus_meta.cc:287] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.571s user 0.005s sys 0.009s
I20251026 14:58:54.000946 23657 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.031224 23661 raft_consensus.cc:3060] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.057385 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:54.037992 23600 raft_consensus.cc:740] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:54.037441 23634 consensus_queue.cc:1059] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20251026 14:58:54.051018 23633 raft_consensus.cc:3055] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
W20251026 14:58:54.065840 22697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:54.064576 22700 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:58:54.078970 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:54.059854 23582 raft_consensus.cc:3055] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:54.050743 23658 consensus_queue.cc:260] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:58:54.055157 23625 consensus_queue.cc:1059] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
W20251026 14:58:54.087757 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:54.090466 23402 raft_consensus.cc:1240] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:54.068753 23616 raft_consensus.cc:2749] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:54.092757 22700 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:54.091401 23654 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:54.091557 23605 consensus_queue.cc:1048] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20251026 14:58:54.097937 23678 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:54.070773 23633 raft_consensus.cc:740] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:54.063416 23652 consensus_queue.cc:1059] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20251026 14:58:54.061466 23660 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.126222 22700 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:54.084717 23582 raft_consensus.cc:740] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:54.143857 22733 raft_consensus.cc:1275] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251026 14:58:54.088935 23653 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:54.127943 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:54.077971 23651 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:54.135177 22731 raft_consensus.cc:1275] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251026 14:58:54.135481 22728 raft_consensus.cc:1275] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251026 14:58:54.098582 23617 consensus_queue.cc:1048] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20251026 14:58:54.150218 23616 raft_consensus.cc:2749] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:54.098606 23607 consensus_queue.cc:1059] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20251026 14:58:54.136538 23680 raft_consensus.cc:2749] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:54.138065 23608 raft_consensus.cc:3055] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:54.152446 23608 raft_consensus.cc:740] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:54.139772 22657 raft_consensus.cc:1275] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251026 14:58:54.131143 23602 raft_consensus.cc:2749] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:54.148151 22700 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:54.153507 23608 consensus_queue.cc:260] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.133049 23654 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.154765 23608 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.155428 23602 raft_consensus.cc:2804] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:54.155808 23602 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:54.156029 23654 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:54.156167 23602 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.154695 22700 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:54.131752 23600 consensus_queue.cc:260] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.157696 23369 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:54.158325 23605 consensus_queue.cc:1048] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 14:58:54.161793 22700 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:54.163302 22700 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:54.163650 22652 raft_consensus.cc:1240] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: []
I20251026 14:58:54.140381 23592 raft_consensus.cc:3055] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:54.164516 22700 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:54.167340 22700 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:54.131349 23662 consensus_queue.cc:1059] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20251026 14:58:54.176105 22700 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:54.162683 23614 raft_consensus.cc:2749] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:54.178776 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:54.177795 22700 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:54.139746 23665 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.163359 23617 raft_consensus.cc:2749] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:54.165122 23654 raft_consensus.cc:2749] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:54.166374 23611 raft_consensus.cc:2804] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:54.186532 22652 raft_consensus.cc:1240] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:54.168891 23615 raft_consensus.cc:3055] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:54.168512 23616 raft_consensus.cc:2749] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:54.166541 23680 raft_consensus.cc:2749] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:54.200820 22624 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:54.167649 23592 raft_consensus.cc:740] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:54.220276 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:54.164804 23653 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.165216 23651 consensus_queue.cc:1059] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
W20251026 14:58:54.237468 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:54.238962 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.239584 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.240000 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.235868 22624 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:54.130812 23678 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.243340 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:54.244288 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:54.247133 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.182710 23600 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.196542 23664 raft_consensus.cc:2749] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:54.180509 23652 raft_consensus.cc:2749] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:54.232759 23611 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:54.263794 23677 raft_consensus.cc:3055] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:54.264396 23677 raft_consensus.cc:740] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:54.263922 23611 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.263782 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.267499 22732 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.252352 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.243366 23647 raft_consensus.cc:2804] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:54.257472 23657 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:54.257638 23658 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.259572 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.233429 23615 raft_consensus.cc:740] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:54.254556 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.272049 22725 raft_consensus.cc:3060] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.183847 23641 raft_consensus.cc:697] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:54.274446 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.277770 23676 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:54.249150 23625 raft_consensus.cc:2804] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:54.282429 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.244408 23617 consensus_queue.cc:1048] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20251026 14:58:54.269593 23647 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:54.200477 23605 consensus_queue.cc:1048] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.089s
I20251026 14:58:54.303826 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.306954 22733 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.269361 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.279769 23667 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.338282 23665 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:54.290181 23625 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:54.340444 23625 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.341074 23678 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:54.299178 23640 raft_consensus.cc:3055] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:54.331904 22725 raft_consensus.cc:2468] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:54.334187 23687 ts_tablet_manager.cc:933] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1: Initiating tablet copy from peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:54.312712 23608 raft_consensus.cc:493] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:54.202384 23633 consensus_queue.cc:260] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.245326 23582 consensus_queue.cc:260] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.278738 23670 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:54.265576 23677 consensus_queue.cc:260] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.301823 22657 raft_consensus.cc:1275] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 14:58:54.351550 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.307457 23647 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.383019 23608 raft_consensus.cc:515] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:54.384891 23678 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:54.338346 23653 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:54.344266 23660 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:54.400202 22734 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:54.344063 23640 raft_consensus.cc:740] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:54.416635 22733 raft_consensus.cc:2468] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:54.401510 23678 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.423178 23678 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:54.425226 23608 leader_election.cc:290] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:54.426563 22731 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.422770 23640 consensus_queue.cc:260] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:54.433249 22729 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:54.435058 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:54.435912 22729 raft_consensus.cc:2468] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:58:54.431615 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.361562 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.441069 22725 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.443974 22732 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:54.446362 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.447077 22734 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.407212 23582 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.460711 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.476605 22732 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.448283 23633 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.360188 23615 consensus_queue.cc:260] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.463274 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:54.408479 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.471138 22731 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:54.472925 22733 raft_consensus.cc:3060] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.495021 22729 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:54.497959 23611 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.499490 23677 raft_consensus.cc:3060] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.502000 22730 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:54.500026 23611 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:54.486346 23592 consensus_queue.cc:260] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:54.496990 22732 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:54.507771 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:54.510102 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:54.510790 22732 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:54.514418 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.505004 22727 raft_consensus.cc:3060] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.518709 23602 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.530362 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:54.533178 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.511809 23640 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.535948 22733 raft_consensus.cc:2468] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:54.466652 23641 consensus_queue.cc:237] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:54.548123 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:54.535739 22725 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:54.557296 23677 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:54.559160 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.559243 23677 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.556008 22726 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
W20251026 14:58:54.550670 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 316879us
I20251026 14:58:54.558403 22734 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:54.570858 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:54.573788 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:54.574764 22624 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:54.579298 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:54.580632 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:54.559865 22732 raft_consensus.cc:2376] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2. Re-sending same reply.
I20251026 14:58:54.582461 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:54.584540 22624 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:54.586910 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:54.590409 23677 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:54.592638 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:54.595364 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.595582 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:54.596107 22732 raft_consensus.cc:2376] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3. Re-sending same reply.
I20251026 14:58:54.596251 22651 raft_consensus.cc:2393] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 2: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:58:54.597189 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:54.597987 22725 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:54.599316 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:54.600064 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:54.600104 22733 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:54.600013 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:54.600867 22732 raft_consensus.cc:2468] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:54.602054 22658 raft_consensus.cc:1240] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
W20251026 14:58:54.558707 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 380729us
I20251026 14:58:54.570160 23662 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:54.603824 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:54.612793 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:54.616839 22729 raft_consensus.cc:2468] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
W20251026 14:58:54.611909 22700 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:54.605654 23623 consensus_queue.cc:1048] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.200s
I20251026 14:58:54.591181 23591 raft_consensus.cc:2749] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:58:54.807410 23695 consensus_queue.cc:1048] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20251026 14:58:54.691555 22774 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.015s sys 0.019s
W20251026 14:58:54.809489 22774 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.747s user 0.018s sys 0.024s
W20251026 14:58:54.809847 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 774617us
I20251026 14:58:54.810310 23695 consensus_queue.cc:1059] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:54.612116 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:54.628376 23592 raft_consensus.cc:3060] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.812105 23698 consensus_queue.cc:1059] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:54.811910 22732 raft_consensus.cc:2376] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2. Re-sending same reply.
I20251026 14:58:54.813189 23698 consensus_queue.cc:1059] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:54.813555 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.814314 22732 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.609040 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:58:54.596634 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 437547us
I20251026 14:58:54.817757 23694 consensus_queue.cc:1059] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:54.817791 23641 consensus_queue.cc:1059] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:58:54.609882 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.818789 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.819108 22726 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:54.818789 23694 raft_consensus.cc:3055] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:54.819592 22725 raft_consensus.cc:3060] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.819725 23694 raft_consensus.cc:740] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:54.820550 23694 consensus_queue.cc:260] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.821718 23694 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.823256 23611 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:54.819077 23696 raft_consensus.cc:3055] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:54.824503 23696 raft_consensus.cc:740] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:54.824237 23611 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:54.825380 23696 consensus_queue.cc:260] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.826228 22726 raft_consensus.cc:2468] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:58:54.826426 23611 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:54.826543 23696 raft_consensus.cc:3060] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.826141 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:54.827423 22650 raft_consensus.cc:2393] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 2: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:58:54.827688 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.828269 22729 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.829255 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:54.830181 22726 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:54.830941 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:54.831461 22726 raft_consensus.cc:2376] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2. Re-sending same reply.
I20251026 14:58:54.832196 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.760511 23625 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.834832 23625 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:54.751757 22731 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:54.761651 22730 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
W20251026 14:58:54.580850 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 421118us
I20251026 14:58:54.836788 23677 raft_consensus.cc:3055] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:54.837179 23677 raft_consensus.cc:740] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:54.761736 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:54.838935 23404 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:58:54.839509 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.840023 22733 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.838114 23677 consensus_queue.cc:260] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.841457 23677 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.842708 22806 raft_consensus.cc:1275] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251026 14:58:54.604310 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:54.845302 22552 catalog_manager.cc:5649] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: term changed from 1 to 2, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4). New cstate: current_term: 2 leader_uuid: "248493ec312e4398b828c6cb00e5c4b1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 14:58:54.590467 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:54.630297 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 57525.1us
I20251026 14:58:54.847680 22552 catalog_manager.cc:5649] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: term changed from 1 to 2, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4). New cstate: current_term: 2 leader_uuid: "248493ec312e4398b828c6cb00e5c4b1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 14:58:54.610674 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:54.766374 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:54.767390 23602 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:54.749572 22727 raft_consensus.cc:2468] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:54.772184 23660 raft_consensus.cc:2749] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:58:54.775863 23400 raft_consensus.cc:1240] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:54.852492 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:58:54.779186 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:54.853379 22727 raft_consensus.cc:2393] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 2: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:58:54.784690 22554 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38884 (request call id 152) took 1085 ms. Trace:
I20251026 14:58:54.854264 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:54.855211 22727 raft_consensus.cc:2393] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 2: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:58:54.853803 22554 rpcz_store.cc:276] 1026 14:58:53.699319 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:53.699580 (+ 261us) service_pool.cc:225] Handling call
1026 14:58:54.784589 (+1085009us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 14:58:53.962790 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 14:58:53.963424 (+ 634us) write_op.cc:432] Acquiring schema lock in shared mode
1026 14:58:53.963465 (+ 41us) write_op.cc:435] Acquired schema lock
1026 14:58:53.963481 (+ 16us) tablet.cc:660] Decoding operations
1026 14:58:53.963899 (+ 418us) write_op.cc:620] Acquiring the partition lock for write op
1026 14:58:53.963966 (+ 67us) write_op.cc:641] Partition lock acquired for write op
1026 14:58:53.963993 (+ 27us) tablet.cc:683] Acquiring locks for 1 operations
1026 14:58:53.964197 (+ 204us) tablet.cc:699] Row locks acquired
1026 14:58:53.964210 (+ 13us) write_op.cc:260] PREPARE: finished
1026 14:58:53.964372 (+ 162us) write_op.cc:270] Start()
1026 14:58:53.964486 (+ 114us) write_op.cc:276] Timestamp: P: 1761490733964317 usec, L: 0
1026 14:58:53.964503 (+ 17us) op_driver.cc:348] REPLICATION: starting
1026 14:58:53.965259 (+ 756us) log.cc:844] Serialized 526 byte log entry
1026 14:58:54.776223 (+810964us) op_driver.cc:464] REPLICATION: finished
1026 14:58:54.778024 (+ 1801us) write_op.cc:301] APPLY: starting
1026 14:58:54.778126 (+ 102us) tablet.cc:1365] starting BulkCheckPresence
1026 14:58:54.778255 (+ 129us) tablet.cc:1368] finished BulkCheckPresence
1026 14:58:54.778419 (+ 164us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 14:58:54.781706 (+ 3287us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 14:58:54.781878 (+ 172us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1026 14:58:54.781930 (+ 52us) write_op.cc:312] APPLY: finished
1026 14:58:54.782127 (+ 197us) log.cc:844] Serialized 22 byte log entry
1026 14:58:54.783192 (+ 1065us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:58:54.783276 (+ 84us) write_op.cc:454] Released schema lock
1026 14:58:54.783325 (+ 49us) write_op.cc:341] FINISH: Updating metrics
1026 14:58:54.788855 (+ 5530us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:58:54.788898 (+ 43us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":1499,"apply.run_cpu_time_us":1871,"apply.run_wall_time_us":7463,"num_ops":1,"prepare.queue_time_us":226825,"prepare.run_cpu_time_us":16102,"prepare.run_wall_time_us":110465,"raft.queue_time_us":598036,"raft.run_cpu_time_us":1476,"raft.run_wall_time_us":13419,"replication_time_us":811493,"thread_start_us":639751,"threads_started":4,"wal-append.queue_time_us":92222}]]}
W20251026 14:58:54.789438 23661 consensus_meta.cc:287] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.692s user 0.008s sys 0.004s
I20251026 14:58:54.792448 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:54.861912 22728 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.851462 22552 catalog_manager.cc:5649] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: term changed from 1 to 2, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4). New cstate: current_term: 2 leader_uuid: "248493ec312e4398b828c6cb00e5c4b1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 14:58:54.793612 22799 raft_consensus.cc:1240] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
W20251026 14:58:54.794004 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 185065us
W20251026 14:58:54.602476 23597 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.003s sys 0.014s
I20251026 14:58:54.795248 23687 tablet_copy_client.cc:323] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.2.2:46507
W20251026 14:58:54.795943 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 211513us
I20251026 14:58:54.885042 22624 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:54.796388 23662 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:54.799786 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:54.799693 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:58:54.898892 22732 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:54.834964 23615 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.814172 23698 consensus_queue.cc:1059] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:54.815289 23699 raft_consensus.cc:3055] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:54.816181 23608 raft_consensus.cc:3055] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:54.817057 22774 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:58:54.756538 23666 consensus_meta.cc:287] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.783s user 0.022s sys 0.005s
I20251026 14:58:54.816949 23686 consensus_queue.cc:1059] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:58:54.834887 22729 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:54.878336 22725 raft_consensus.cc:2468] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:54.882706 22726 raft_consensus.cc:2468] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:54.785037 23647 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.902243 22624 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:58:54.883661 23597 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.817s user 0.006s sys 0.015s
I20251026 14:58:54.907181 22733 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:54.914724 22734 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:54.903741 23640 raft_consensus.cc:2749] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:58:54.906014 23633 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:54.908510 23699 raft_consensus.cc:740] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:54.910261 23608 raft_consensus.cc:740] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:54.927405 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:54.921746 23592 raft_consensus.cc:2749] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:58:54.931113 23689 consensus_queue.cc:1048] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20251026 14:58:54.942203 23684 raft_consensus.cc:3055] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:54.942653 23684 raft_consensus.cc:740] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:54.943694 22744 tablet_copy_service.cc:140] P 562bb1e1c8e445b8baecd29c3669658e: Received BeginTabletCopySession request for tablet 2efb26bccebf43b5b8a85f51fd977714 from peer 248493ec312e4398b828c6cb00e5c4b1 ({username='slave'} at 127.0.0.1:47996)
I20251026 14:58:54.944424 23699 consensus_queue.cc:260] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.944417 22744 tablet_copy_service.cc:161] P 562bb1e1c8e445b8baecd29c3669658e: Beginning new tablet copy session on tablet 2efb26bccebf43b5b8a85f51fd977714 from peer 248493ec312e4398b828c6cb00e5c4b1 at {username='slave'} at 127.0.0.1:47996: session id = 248493ec312e4398b828c6cb00e5c4b1-2efb26bccebf43b5b8a85f51fd977714
I20251026 14:58:54.946158 23699 raft_consensus.cc:3060] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.926205 23633 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:54.950766 23633 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:54.950965 23695 raft_consensus.cc:2955] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:54.954442 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:58:54.957142 23684 consensus_queue.cc:260] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.936363 23607 raft_consensus.cc:2804] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:54.941627 23608 consensus_queue.cc:260] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:54.957190 22700 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:54.965173 23607 raft_consensus.cc:697] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:54.970974 23607 consensus_queue.cc:237] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.958442 22650 raft_consensus.cc:2955] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:58:54.971290 23647 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:54.986900 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:54.973060 22728 raft_consensus.cc:2468] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:55.004159 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:55.005007 22802 raft_consensus.cc:2393] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:54.974865 23608 raft_consensus.cc:3060] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:54.987874 23662 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:54.995601 23702 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:55.024857 23702 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:54.973457 23684 raft_consensus.cc:3060] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
W20251026 14:58:55.029698 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 115830us
W20251026 14:58:55.030081 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 213089us
I20251026 14:58:55.042032 22773 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.043969 23720 raft_consensus.cc:2749] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.044703 22773 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:58:55.053558 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 151394us
I20251026 14:58:55.050830 22773 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.055074 22773 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.055964 22773 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.056830 22773 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.058584 23713 raft_consensus.cc:2749] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.059320 23698 consensus_queue.cc:1059] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:55.059654 23597 raft_consensus.cc:3055] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:55.060132 23597 raft_consensus.cc:740] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:55.060997 23597 consensus_queue.cc:260] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.061920 23686 raft_consensus.cc:2749] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.062115 23696 raft_consensus.cc:2749] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.035452 23710 consensus_queue.cc:1059] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20251026 14:58:55.063318 23710 raft_consensus.cc:3055] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:55.063648 23710 raft_consensus.cc:740] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:55.064489 22773 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.064494 23710 consensus_queue.cc:260] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.065977 23710 raft_consensus.cc:3060] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:55.066123 22773 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.067235 23698 raft_consensus.cc:2749] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.068280 23698 raft_consensus.cc:2749] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.068553 23699 raft_consensus.cc:3055] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:55.068939 23699 raft_consensus.cc:740] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:55.061826 23694 raft_consensus.cc:2749] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.069545 22773 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.070675 22773 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.074753 22624 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:55.078267 23662 raft_consensus.cc:2804] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:55.079016 23662 raft_consensus.cc:697] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:55.079631 22624 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:55.081720 23660 raft_consensus.cc:2749] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.080312 23662 consensus_queue.cc:237] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.083552 22624 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:55.069761 23699 consensus_queue.cc:260] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.075904 23694 raft_consensus.cc:2749] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.037515 23633 consensus_queue.cc:1059] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20251026 14:58:55.088325 22624 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:55.088596 22773 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.062508 23597 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:55.104339 22658 raft_consensus.cc:1240] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:55.115054 22651 raft_consensus.cc:1240] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:55.117300 22624 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:55.075038 23720 raft_consensus.cc:2749] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.096843 23607 consensus_queue.cc:1059] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20251026 14:58:55.089068 23694 raft_consensus.cc:2749] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.122080 23647 raft_consensus.cc:2804] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:55.122408 23713 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:55.107554 22655 raft_consensus.cc:1240] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:55.124456 23686 raft_consensus.cc:2749] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.126204 23702 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:55.127820 22773 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.120844 23592 raft_consensus.cc:2804] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:55.121989 22659 raft_consensus.cc:1240] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:55.089011 23625 raft_consensus.cc:2804] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:55.133785 22624 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:55.086179 23699 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:55.109359 22657 raft_consensus.cc:1240] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:55.089092 23711 raft_consensus.cc:3055] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:55.158778 23615 raft_consensus.cc:3055] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:55.158942 23711 raft_consensus.cc:740] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:55.159221 22773 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.160315 23717 raft_consensus.cc:2749] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.160442 22773 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.150223 23647 raft_consensus.cc:697] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:55.159193 23615 raft_consensus.cc:740] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:55.162132 23647 consensus_queue.cc:237] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.162788 23615 consensus_queue.cc:260] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.164307 23615 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:55.120215 23641 consensus_queue.cc:1059] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20251026 14:58:55.150065 23713 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:55.158304 23607 raft_consensus.cc:2749] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.171169 23713 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:55.149533 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:55.176126 23711 consensus_queue.cc:260] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.194754 23711 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:55.197153 22803 raft_consensus.cc:2393] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:55.138883 23582 raft_consensus.cc:3055] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:55.199240 23582 raft_consensus.cc:740] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:55.211794 23582 consensus_queue.cc:260] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.213567 23582 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
W20251026 14:58:55.224704 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 98519.7us
I20251026 14:58:55.226737 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:58:55.227622 23402 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:55.237556 22771 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters:
W20251026 14:58:55.239074 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 71318.7us
W20251026 14:58:55.265023 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 104636us
I20251026 14:58:55.268882 23686 raft_consensus.cc:2749] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.269994 23686 raft_consensus.cc:3055] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:58:55.270439 23686 raft_consensus.cc:740] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:55.271548 23686 consensus_queue.cc:260] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.268074 22773 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.275077 22773 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.276062 23711 raft_consensus.cc:2804] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:55.276563 23711 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:55.277158 23711 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:55.277524 23720 raft_consensus.cc:2804] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:55.278345 23684 raft_consensus.cc:2749] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.279371 23720 raft_consensus.cc:697] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:55.280668 23720 consensus_queue.cc:237] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:55.282140 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:55.284179 22773 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.285794 23710 raft_consensus.cc:2749] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.286293 22773 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.288424 23686 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:55.260577 22554 catalog_manager.cc:5649] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 1 to 2, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 2 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:55.296600 23695 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:47990 (request call id 2) took 2603 ms. Trace:
I20251026 14:58:55.297168 23695 rpcz_store.cc:276] 1026 14:58:52.692822 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:52.693372 (+ 550us) service_pool.cc:225] Handling call
1026 14:58:52.889319 (+195947us) log.cc:844] Serialized 485 byte log entry
1026 14:58:53.135570 (+246251us) spinlock_profiling.cc:243] Waited 2.99 ms on lock 0x7b180032bc00. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce026caa 00007fd8d7698a29 00007fd8cdee6c30 00007fd8ce0aea94 00007fd8ce0aea05 00007fd8d10b7a5b 00007fd8d113c7b3 00007fd8d11642e2 00007fd8d61082e9
1026 14:58:55.296488 (+2160918us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":155842,"raft.queue_time_us":345831,"raft.run_cpu_time_us":2092,"raft.run_wall_time_us":22151,"spinlock_wait_cycles":6886272,"thread_start_us":177339,"threads_started":2,"wal-append.queue_time_us":118384,"wal-append.run_cpu_time_us":2309,"wal-append.run_wall_time_us":1054170}
I20251026 14:58:55.304004 22744 tablet_copy_source_session.cc:215] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e: Tablet Copy: opened 0 blocks and 1 log segments
W20251026 14:58:55.305614 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 184473us
I20251026 14:58:55.306417 22624 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:55.306154 22799 consensus_queue.cc:237] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:55.307551 22624 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:55.308737 23695 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:55.309054 23674 raft_consensus.cc:2804] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:55.309523 23674 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:55.309870 23674 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:55.309268 23695 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.310163 23592 raft_consensus.cc:697] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:55.311060 23592 consensus_queue.cc:237] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.313480 23665 raft_consensus.cc:2804] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:55.313910 23702 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:55.314244 23665 raft_consensus.cc:697] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:55.314460 23702 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.315258 23665 consensus_queue.cc:237] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.316455 23702 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:55.317301 23625 raft_consensus.cc:697] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:55.318549 23625 consensus_queue.cc:237] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.321573 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:55.322055 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:55.322429 22806 raft_consensus.cc:2393] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:55.308780 22624 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:55.313551 23695 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:55.324009 22657 raft_consensus.cc:2393] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 2: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:58:55.320065 22773 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.327859 22624 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:55.331985 22773 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.313524 23660 raft_consensus.cc:2764] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20251026 14:58:55.333097 23710 raft_consensus.cc:2804] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:55.333817 23710 raft_consensus.cc:697] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:55.333874 23684 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:58:55.334517 23684 raft_consensus.cc:697] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:55.334847 23710 consensus_queue.cc:237] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.335371 23684 consensus_queue.cc:237] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.339300 22773 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.341302 23713 raft_consensus.cc:2804] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:55.341706 23713 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:55.342078 23713 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:55.319312 23696 raft_consensus.cc:2749] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:58:55.344698 22624 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:55.344846 22773 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.346674 22624 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:55.348639 22624 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:55.350703 22624 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:55.351462 22773 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.355154 22773 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.361068 22773 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.352267 23634 raft_consensus.cc:2804] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:55.377257 22773 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.380254 23701 raft_consensus.cc:2804] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:55.380895 23701 raft_consensus.cc:697] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:55.347631 23660 raft_consensus.cc:2804] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:55.383456 22806 raft_consensus.cc:1240] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:55.349635 23702 raft_consensus.cc:2804] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:55.388540 22733 raft_consensus.cc:1240] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:55.352352 23651 raft_consensus.cc:2804] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:55.390095 22725 raft_consensus.cc:1240] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:55.401033 22725 raft_consensus.cc:1240] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:55.391062 22729 raft_consensus.cc:1240] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:55.349545 23696 raft_consensus.cc:2749] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.392818 22806 raft_consensus.cc:1240] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 2. Current term is 3. Ops: []
I20251026 14:58:55.345547 23724 raft_consensus.cc:2804] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:55.384282 22624 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:55.388053 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:55.403842 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:55.384879 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:55.404712 22731 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:55.405577 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:55.389142 22730 raft_consensus.cc:1240] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:55.384618 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:55.386255 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:55.387776 22734 raft_consensus.cc:1240] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:55.374512 23597 raft_consensus.cc:2749] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.381764 23701 consensus_queue.cc:237] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.403415 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:55.414386 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:55.415035 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:55.409312 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:55.375823 23615 raft_consensus.cc:2804] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:55.415464 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:58:55.418956 22700 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:55.420843 23633 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:55.421044 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:55.421921 22733 raft_consensus.cc:1240] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: []
I20251026 14:58:55.422348 23641 consensus_queue.cc:1059] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:55.422987 23607 consensus_queue.cc:1059] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:55.423802 22773 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:55.405841 23660 raft_consensus.cc:697] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:55.430320 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:55.382929 23582 raft_consensus.cc:2804] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:55.429464 23662 raft_consensus.cc:2804] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:55.430877 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:55.435787 22697 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.434965 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:55.452098 22801 raft_consensus.cc:2468] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:55.438699 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:55.395121 23634 raft_consensus.cc:697] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:55.424368 22807 raft_consensus.cc:1240] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:55.442791 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:55.407078 23702 raft_consensus.cc:697] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:55.395429 23698 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:55.402947 23720 raft_consensus.cc:2804] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:55.403141 23711 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:55.431787 23651 raft_consensus.cc:697] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:55.440163 23615 raft_consensus.cc:697] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:55.432952 23729 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:55.441356 23695 consensus_queue.cc:1059] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20251026 14:58:55.439664 23661 consensus_queue.cc:1059] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20251026 14:58:55.439020 23694 consensus_queue.cc:1059] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20251026 14:58:55.439304 23633 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.489691 23711 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:55.461047 23662 raft_consensus.cc:697] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:55.444528 23674 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.472754 23597 raft_consensus.cc:2749] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:58:55.471038 23717 consensus_queue.cc:1059] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20251026 14:58:55.478621 22803 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:55.482213 23720 raft_consensus.cc:697] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:55.467337 23691 consensus_queue.cc:1059] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20251026 14:58:55.487041 23607 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:55.518510 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:58:55.519340 23402 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Advancing to term 3
W20251026 14:58:55.520020 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 102855us
I20251026 14:58:55.506035 23677 raft_consensus.cc:2749] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.496819 23608 consensus_queue.cc:1059] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20251026 14:58:55.526640 22657 raft_consensus.cc:1240] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:55.512056 23665 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
W20251026 14:58:55.530737 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 146526us
I20251026 14:58:55.479512 23698 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:55.527333 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:55.520016 23592 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:55.536731 22659 raft_consensus.cc:1275] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:55.536831 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:55.500811 23713 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.538739 22624 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:55.559556 22624 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:55.542271 23641 consensus_queue.cc:1059] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:58:55.485338 23729 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.561335 22624 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:55.565546 22624 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:55.521898 23607 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.512321 23660 consensus_queue.cc:237] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:55.579474 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 146070us
I20251026 14:58:55.578017 23625 raft_consensus.cc:2804] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:58:55.579924 22808 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:55.528107 23651 consensus_queue.cc:237] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.580313 22697 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.562173 22650 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:55.587734 22697 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:58:55.589519 23734 log.cc:927] Time spent T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.164s user 0.002s sys 0.000s
I20251026 14:58:55.543840 23717 consensus_queue.cc:1059] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20251026 14:58:55.588858 22697 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.558127 23597 consensus_queue.cc:1048] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20251026 14:58:55.528270 23634 consensus_queue.cc:237] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.557955 23592 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.544209 23665 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.597930 22697 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.538115 23702 consensus_queue.cc:237] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.544600 23615 consensus_queue.cc:237] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:55.615221 23730 log.cc:927] Time spent T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.088s user 0.001s sys 0.001s
I20251026 14:58:55.570860 23662 consensus_queue.cc:237] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.616401 23687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2efb26bccebf43b5b8a85f51fd977714. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:55.582823 23720 consensus_queue.cc:237] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.621865 23674 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:55.622311 23402 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:58:55.622742 23674 raft_consensus.cc:2804] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:55.623268 23674 raft_consensus.cc:697] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:55.617339 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:55.613875 23736 raft_consensus.cc:2804] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election won for term 2
W20251026 14:58:55.673146 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 215261us
I20251026 14:58:55.684914 22773 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.625730 23625 raft_consensus.cc:697] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:55.654937 23695 raft_consensus.cc:2749] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:58:55.658430 23607 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:55.657220 23592 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:55.657050 23746 raft_consensus.cc:2749] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.657114 23738 raft_consensus.cc:2804] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election won for term 2
W20251026 14:58:55.694407 23699 consensus_meta.cc:287] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.500s user 0.012s sys 0.000s
I20251026 14:58:55.630285 23633 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:55.678860 23582 raft_consensus.cc:697] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:55.681636 22655 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:55.747448 23695 raft_consensus.cc:2749] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.694851 23740 raft_consensus.cc:2749] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.700510 23736 raft_consensus.cc:697] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:55.725925 22807 raft_consensus.cc:1240] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:55.746402 23713 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:55.765064 23647 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:55.769373 23740 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:55.647503 23674 consensus_queue.cc:237] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.776153 23582 consensus_queue.cc:237] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:58:55.777172 23737 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:58:55.777617 23661 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:55.752947 23698 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:55.767675 23724 raft_consensus.cc:697] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:58:55.772986 23735 log.cc:927] Time spent T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.215s user 0.001s sys 0.000s
I20251026 14:58:55.766645 23729 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:55.782560 23736 consensus_queue.cc:237] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:55.802793 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:55.780165 23665 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:55.770920 23625 consensus_queue.cc:237] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.790395 23661 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:55.836618 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 271141us
I20251026 14:58:55.781663 23647 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:55.841231 22624 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:55.873538 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:58:55.874897 23686 consensus_meta.cc:287] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.585s user 0.013s sys 0.000s
I20251026 14:58:55.800199 23740 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.821564 23737 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:58:55.913796 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 72621.4us
I20251026 14:58:55.916512 22624 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:55.904492 23651 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
W20251026 14:58:55.917778 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 291724us
I20251026 14:58:55.924708 23747 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:55.926342 22624 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
W20251026 14:58:55.936723 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 338816us
I20251026 14:58:55.906021 22727 raft_consensus.cc:2376] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2. Re-sending same reply.
I20251026 14:58:55.937829 22697 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.915889 23665 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:55.917287 23701 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
W20251026 14:58:55.940145 22700 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:55.957626 22697 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:55.925530 23695 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:55.967815 22553 catalog_manager.cc:5649] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 1 to 2, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 2 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:55.924710 23724 consensus_queue.cc:237] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.938406 23754 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:55.957196 23640 raft_consensus.cc:2804] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:55.960633 23702 raft_consensus.cc:2804] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:55.926126 23651 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.974917 23695 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.958655 23665 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:58:55.963533 23701 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:55.987715 23640 raft_consensus.cc:697] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:56.009536 23756 raft_consensus.cc:2749] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.998497 23662 raft_consensus.cc:2804] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:56.011509 23753 raft_consensus.cc:2749] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:55.990111 23702 raft_consensus.cc:697] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:55.939002 23747 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.006680 23647 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:55.981027 23754 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.017642 23661 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:56.019383 23662 raft_consensus.cc:697] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:56.030012 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:56.035985 22651 raft_consensus.cc:2468] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:56.035764 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:56.023135 23651 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:58:56.041476 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 115246us
I20251026 14:58:56.030869 23756 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:56.067121 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:56.068704 23737 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:56.076287 23662 consensus_queue.cc:237] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:56.086694 23758 log.cc:927] Time spent T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20251026 14:58:56.072173 23756 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.094668 23756 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:56.071301 23640 consensus_queue.cc:237] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.072497 23702 consensus_queue.cc:237] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.085754 23740 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:56.099802 23695 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:56.102141 23701 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:56.097649 23647 consensus_queue.cc:1059] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20251026 14:58:56.130550 23754 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:56.111099 23665 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:56.133968 23696 raft_consensus.cc:3055] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
W20251026 14:58:56.128502 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 170906us
I20251026 14:58:56.136425 22697 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:56.140532 22697 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
W20251026 14:58:56.142570 23760 log.cc:927] Time spent T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20251026 14:58:56.141775 23695 raft_consensus.cc:2749] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:58:56.157618 23716 raft_consensus.cc:3055] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:56.158068 23716 raft_consensus.cc:740] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:56.158968 23716 consensus_queue.cc:260] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.161264 23716 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:56.135550 23696 raft_consensus.cc:740] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:56.163385 23696 consensus_queue.cc:260] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.165614 23747 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:56.166741 23740 raft_consensus.cc:2749] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:58:56.168274 22697 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:56.173494 23747 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:56.174193 23747 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.176707 23747 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:56.178102 23677 raft_consensus.cc:3055] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:56.178634 23677 raft_consensus.cc:740] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:56.165005 23696 raft_consensus.cc:3060] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:56.185626 23717 raft_consensus.cc:3055] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:56.186062 23717 raft_consensus.cc:740] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:56.187057 23739 raft_consensus.cc:3055] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:58:56.187247 23701 raft_consensus.cc:3055] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:56.186946 23717 consensus_queue.cc:260] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.187680 23739 raft_consensus.cc:740] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:56.187747 23701 raft_consensus.cc:740] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:56.190280 23701 consensus_queue.cc:260] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:56.191992 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 507090us
I20251026 14:58:56.192237 23597 raft_consensus.cc:2749] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:58:56.193724 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:58:56.179723 23677 consensus_queue.cc:260] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:56.194096 22773 leader_election.cc:304] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:56.194440 23400 raft_consensus.cc:2468] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:56.194808 23677 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:56.196027 23694 raft_consensus.cc:3055] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:56.196452 23694 raft_consensus.cc:740] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:56.197916 23699 raft_consensus.cc:2749] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:58:56.197547 23607 consensus_queue.cc:1059] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:58:56.198702 23684 raft_consensus.cc:3055] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:56.199082 23684 raft_consensus.cc:740] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:56.197219 23694 consensus_queue.cc:260] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.188898 23739 consensus_queue.cc:260] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.200284 23694 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:56.200520 23739 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:56.201771 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:56.202467 22653 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
W20251026 14:58:56.217988 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 49827us
I20251026 14:58:56.219584 22697 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:56.227037 22697 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:56.244392 22697 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:56.225306 23740 raft_consensus.cc:2749] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:56.218786 23695 raft_consensus.cc:2804] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:56.246501 23695 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:56.246918 23695 raft_consensus.cc:3060] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:56.242842 23753 raft_consensus.cc:2764] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20251026 14:58:56.221603 23633 consensus_queue.cc:1059] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20251026 14:58:56.252409 23740 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:56.243081 23747 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:56.255124 23753 raft_consensus.cc:2804] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:56.269377 23720 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:56.269912 23720 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.271245 22802 raft_consensus.cc:1240] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: [1.2-1.2]
I20251026 14:58:56.271857 23720 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:56.272300 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:56.272446 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:56.273058 22805 raft_consensus.cc:2468] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:56.273556 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:56.274153 22804 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:56.274190 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:56.274873 22806 raft_consensus.cc:2468] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:56.275187 22801 raft_consensus.cc:1240] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 2. Current term is 3. Ops: []
I20251026 14:58:56.275823 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:56.276433 22800 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:56.276841 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:56.277508 22799 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:56.260497 23746 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:56.288550 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:56.298206 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:56.299137 22657 raft_consensus.cc:2393] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 2: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:58:56.298226 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:56.279351 22552 catalog_manager.cc:5649] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New cstate: current_term: 1 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 14:58:56.261816 23756 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:56.264808 23661 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:56.305161 22697 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:56.306326 22697 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:56.307420 22697 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:56.260856 23740 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.278465 23753 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:56.312108 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:56.263810 23747 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.304445 23592 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:56.300439 22650 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:56.308830 22697 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:56.335299 22655 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:56.320837 23753 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:56.343945 23767 raft_consensus.cc:2804] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:56.344487 23767 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:56.344858 23767 raft_consensus.cc:3060] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
W20251026 14:58:56.314741 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:56.323935 23755 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:56.315115 23661 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.325449 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:56.361137 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 306683us
I20251026 14:58:56.314522 23756 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.303156 23746 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.378068 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:56.378755 22806 raft_consensus.cc:1275] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:56.380828 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:58:56.382400 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:56.384250 22774 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:56.390487 22774 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:56.391846 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:56.392357 22774 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:56.393378 22805 raft_consensus.cc:2393] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:56.392920 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:56.394706 22650 raft_consensus.cc:3060] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:56.395334 22801 raft_consensus.cc:1275] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:56.405958 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:56.406898 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:56.393003 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:56.408820 22650 raft_consensus.cc:1275] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:56.359499 23738 raft_consensus.cc:697] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:58:56.412503 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:56.414582 22652 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:56.342384 23592 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:56.416731 22654 raft_consensus.cc:1275] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:56.407356 23696 raft_consensus.cc:2804] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:56.420110 23765 raft_consensus.cc:2804] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:56.409929 23607 raft_consensus.cc:2804] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:56.359795 23767 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.410364 23641 raft_consensus.cc:2804] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:56.414589 23762 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:56.418772 23739 raft_consensus.cc:2804] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:56.465736 23729 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:56.465734 23742 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:56.398845 23755 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.467093 23710 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:56.467361 23749 consensus_queue.cc:1048] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:58:56.467159 23742 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.467856 23710 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.468668 23755 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:56.470146 22771 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters: df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:58:56.473698 22771 consensus_peers.cc:597] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 -> Peer 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341): Couldn't send request to peer 248493ec312e4398b828c6cb00e5c4b1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9339edd1ecbf42e3a5e953b2b47f52f2. This is attempt 1: this message will repeat every 5th retry.
I20251026 14:58:56.448406 23696 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:56.463091 22655 raft_consensus.cc:1240] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:56.450376 23607 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:58:56.485656 22553 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.028s sys 0.040s
I20251026 14:58:56.465052 23765 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:56.487331 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:56.488636 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:58:56.489710 23661 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:56.464363 23739 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:56.529559 23739 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:56.453428 23766 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:58:56.486115 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 177354us
I20251026 14:58:56.536121 22697 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:58:56.491747 22553 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.517s user 0.029s sys 0.040s
I20251026 14:58:56.454010 23641 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:56.450363 23691 consensus_queue.cc:1048] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20251026 14:58:56.493605 23607 raft_consensus.cc:3060] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:56.521576 23765 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:56.553650 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:56.526005 23746 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:56.490960 23742 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:56.522652 23674 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:56.527134 23736 consensus_queue.cc:1048] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:56.466573 23729 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.534196 23582 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:56.482714 23720 raft_consensus.cc:2804] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:58:56.492241 23710 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:56.484368 23696 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:56.470813 23762 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.546572 22806 raft_consensus.cc:1275] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:56.473537 23684 consensus_queue.cc:260] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:56.549783 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:56.551584 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:56.538988 23641 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:56.556445 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:56.526373 23661 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:56.556464 22652 raft_consensus.cc:1275] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:58:56.557058 22734 raft_consensus.cc:1275] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 14:58:56.558969 22726 raft_consensus.cc:1275] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:58:56.561580 22730 raft_consensus.cc:1275] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:56.562042 22732 raft_consensus.cc:1275] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:56.549676 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:56.555877 23615 consensus_queue.cc:1048] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20251026 14:58:56.785807 23783 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:56.640022 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:58:56.620651 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 57964us
I20251026 14:58:56.840447 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:56.609447 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:56.876945 22733 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:56.877391 22731 raft_consensus.cc:1275] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:56.712251 22651 raft_consensus.cc:3060] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:56.838999 23783 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:56.839597 23772 log.cc:927] Time spent T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.317s user 0.000s sys 0.000s
W20251026 14:58:56.647580 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 110552us
I20251026 14:58:56.618786 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:57.258550 22729 raft_consensus.cc:2468] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:57.015573 22733 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:56.877403 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:56.565115 23746 raft_consensus.cc:2804] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:56.877735 22800 raft_consensus.cc:2393] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:57.260710 22730 raft_consensus.cc:2468] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:56.559352 23740 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:56.554605 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
W20251026 14:58:57.079047 23695 consensus_meta.cc:287] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.832s user 0.007s sys 0.005s
I20251026 14:58:57.290863 23789 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:58:57.292423 23753 consensus_meta.cc:287] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.918s user 0.011s sys 0.000s
I20251026 14:58:57.033737 23633 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:58:57.215710 23607 consensus_meta.cc:287] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.662s user 0.005s sys 0.004s
W20251026 14:58:57.194075 23779 log.cc:927] Time spent T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.172s user 0.003s sys 0.002s
W20251026 14:58:57.074726 23781 log.cc:927] Time spent T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
W20251026 14:58:57.025364 23782 log.cc:927] Time spent T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20251026 14:58:57.302248 23695 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:57.235304 23694 consensus_meta.cc:287] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 1.035s user 0.011s sys 0.001s
W20251026 14:58:56.992590 23777 log.cc:927] Time spent T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.205s user 0.000s sys 0.000s
W20251026 14:58:57.227643 23677 consensus_meta.cc:287] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 1.032s user 0.011s sys 0.002s
W20251026 14:58:57.025936 23776 log.cc:927] Time spent T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.176s user 0.000s sys 0.001s
I20251026 14:58:56.592065 23674 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
W20251026 14:58:56.937966 23771 log.cc:927] Time spent T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
W20251026 14:58:57.010658 23773 log.cc:927] Time spent T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.463s user 0.000s sys 0.001s
W20251026 14:58:57.100814 23687 consensus_meta.cc:287] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1: Time spent flushing consensus metadata: real 0.930s user 0.008s sys 0.004s
W20251026 14:58:57.016382 23778 log.cc:927] Time spent T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.450s user 0.001s sys 0.001s
I20251026 14:58:56.723603 23625 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:57.028299 22802 raft_consensus.cc:1275] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:56.836727 23701 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:57.203701 23698 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:56.674033 23691 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:57.018042 23755 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:57.151997 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:57.253134 23756 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:57.302475 22729 raft_consensus.cc:1275] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251026 14:58:57.026787 23716 consensus_meta.cc:287] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.865s user 0.011s sys 0.000s
I20251026 14:58:57.069797 23790 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:56.643352 23747 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:56.786051 23592 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:56.911433 23720 raft_consensus.cc:697] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:57.098690 23785 raft_consensus.cc:2804] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:58:57.087438 23615 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:57.276003 23746 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:56.927292 23767 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:56.988168 23762 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:56.936022 23787 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:56.609193 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:56.555930 23766 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:57.321774 22805 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.680s user 0.002s sys 0.000s
W20251026 14:58:57.414510 22805 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.773s user 0.002s sys 0.001s
I20251026 14:58:57.370044 23747 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:57.296355 23717 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:57.326917 22651 raft_consensus.cc:1275] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:57.281656 23582 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.287622 23661 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.302039 23400 raft_consensus.cc:2468] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:57.293901 22727 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
W20251026 14:58:57.331162 23739 consensus_meta.cc:287] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.801s user 0.013s sys 0.001s
I20251026 14:58:56.963364 23784 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:58:57.428612 22726 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:57.331483 23677 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:58:57.341226 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:57.319772 23633 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.318861 23607 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:57.342423 23694 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:57.367934 23791 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:57.368409 23729 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:57.368618 23780 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:57.357746 23597 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:57.360770 23710 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:57.315034 23753 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.369261 22804 raft_consensus.cc:1275] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251026 14:58:57.377614 22624 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.013s sys 0.023s
W20251026 14:58:57.454105 22624 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.608s user 0.014s sys 0.024s
W20251026 14:58:57.454430 22700 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.007s sys 0.005s
W20251026 14:58:57.454823 22700 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.579s user 0.007s sys 0.005s
I20251026 14:58:57.364032 23756 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:56.562814 23699 consensus_queue.cc:1059] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.739s
I20251026 14:58:57.370653 23716 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:57.300782 23789 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.346149 23625 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.350545 23698 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.368443 23795 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:58:57.373008 22773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.008s sys 0.030s
I20251026 14:58:57.361092 23755 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.405503 23746 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:57.409380 23785 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:56.560561 23738 consensus_queue.cc:237] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.411022 22799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44594 (request call id 534) took 1033 ms. Trace:
I20251026 14:58:57.431874 22658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 898) took 1397 ms. Trace:
W20251026 14:58:57.428283 22623 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.010s sys 0.024s
W20251026 14:58:57.434839 22774 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.011s sys 0.020s
W20251026 14:58:57.416960 23370 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.693s user 0.010s sys 0.041s
I20251026 14:58:57.429637 23695 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:57.406917 23615 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.359200 23691 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.425225 23787 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.423657 23747 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.369794 23790 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.462982 22808 raft_consensus.cc:2393] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
W20251026 14:58:57.463601 22622 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.000s sys 0.021s
W20251026 14:58:57.455143 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 579032us
I20251026 14:58:57.430598 23739 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.440021 23784 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.457057 23742 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:58:57.458339 23765 consensus_meta.cc:287] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.905s user 0.014s sys 0.000s
I20251026 14:58:57.453899 23674 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:57.438253 23677 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:58:57.446305 23694 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.455121 23796 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:57.503453 22725 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
W20251026 14:58:57.635994 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:57.457785 23687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:40914 (request call id 38) took 3375 ms. Trace:
I20251026 14:58:57.457073 23729 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:57.500869 23633 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:58:57.676360 23806 log.cc:927] Time spent T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.137s user 0.001s sys 0.000s
I20251026 14:58:57.674010 23767 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:58:57.673117 22623 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.797s user 0.014s sys 0.033s
W20251026 14:58:57.675909 22774 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.760s user 0.011s sys 0.032s
I20251026 14:58:57.712054 22808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44594 (request call id 538) took 1327 ms. Trace:
W20251026 14:58:57.672277 22622 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.827s user 0.006s sys 0.026s
I20251026 14:58:57.680663 22804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 556) took 1127 ms. Trace:
W20251026 14:58:57.675684 22773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.828s user 0.008s sys 0.036s
W20251026 14:58:57.672901 23370 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.964s user 0.015s sys 0.050s
I20251026 14:58:57.633247 23807 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:57.697615 22700 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:57.459875 23597 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.712409 22808 rpcz_store.cc:276] 1026 14:58:56.378370 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:56.379661 (+ 1291us) service_pool.cc:225] Handling call
1026 14:58:57.705834 (+1326173us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54001b7bd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:58:57.705979 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2412116864}
I20251026 14:58:57.460086 23710 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.675570 22658 rpcz_store.cc:276] 1026 14:58:56.028616 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:56.029027 (+ 411us) service_pool.cc:225] Handling call
1026 14:58:56.029494 (+ 467us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:58:56.033906 (+ 4412us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:58:56.033944 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:58:56.036993 (+ 3049us) log.cc:844] Serialized 26 byte log entry
1026 14:58:56.411562 (+374569us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:58:56.411610 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:58:56.412106 (+ 496us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:58:57.426142 (+1014036us) raft_consensus.cc:1685] finished
1026 14:58:57.426180 (+ 38us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:58:57.426349 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3163,"raft.queue_time_us":45262,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":5945,"thread_start_us":340549,"threads_started":1,"wal-append.queue_time_us":380854}
I20251026 14:58:57.680420 23684 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:57.501444 23809 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:57.674926 23785 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:57.637598 23798 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:57.716969 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 920) took 1155 ms. Trace:
I20251026 14:58:57.692498 23695 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:57.713368 22804 rpcz_store.cc:276] 1026 14:58:56.546512 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:56.550182 (+ 3670us) service_pool.cc:225] Handling call
1026 14:58:56.550472 (+ 290us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:58:57.454008 (+903536us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:58:57.673757 (+219749us) spinlock_profiling.cc:243] Waited 815 ms on lock 0x7b540021b6d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:58:57.673961 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1875484032}
I20251026 14:58:57.459534 23791 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.465775 23756 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.472276 23716 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.463765 23720 consensus_queue.cc:237] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:57.674520 22799 rpcz_store.cc:276] 1026 14:58:56.376878 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:56.386292 (+ 9414us) service_pool.cc:225] Handling call
1026 14:58:57.151813 (+765521us) spinlock_profiling.cc:243] Waited 25.8 ms on lock 0x7b5800002a98. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8ce074451 00007fd8ce074302 00007fd8d6f0dad9 00007fd8d6090d30 00007fd8d615984b 00007fd8d61598d1 00007fd8d6107660 00007fd8d6106f44 00007fd8cf6a4834 00007fd8cf6a47c2
1026 14:58:57.410087 (+258274us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":59244928}
I20251026 14:58:57.728426 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:57.686098 23765 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.711074 23674 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:57.751786 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:57.710276 22728 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:57.686286 23742 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.715657 23699 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:57.754869 22725 rpcz_store.cc:276] 1026 14:58:56.555987 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:56.556225 (+ 238us) service_pool.cc:225] Handling call
1026 14:58:57.710395 (+1154170us) spinlock_profiling.cc:243] Waited 629 ms on lock 0x7b54001e8cd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:58:57.710515 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1445742976}
I20251026 14:58:57.759198 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:57.698701 23795 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.761891 23790 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:57.762352 23784 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:57.763041 23789 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:57.763374 23787 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:57.764914 23691 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:57.766018 23615 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:57.699440 23687 rpcz_store.cc:276] 1026 14:58:54.088364 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:54.090806 (+ 2442us) service_pool.cc:225] Handling call
1026 14:58:54.393583 (+302777us) ts_tablet_manager.cc:934] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1: Initiating tablet copy from peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
1026 14:58:55.450417 (+1056834us) tablet_copy_client.cc:368] Tablet copy session begun
1026 14:58:56.169965 (+719548us) tablet_metadata.cc:641] Metadata flushed
1026 14:58:56.170328 (+ 363us) tablet_copy_client.cc:497] Wrote new tablet metadata
1026 14:58:57.344428 (+1174100us) tablet_copy_client.cc:418] Wrote new consensus metadata
1026 14:58:57.418055 (+ 73627us) tablet_replica.cc:206] Creating consensus instance
1026 14:58:57.441035 (+ 22980us) ts_tablet_manager.cc:989] Replica registered.
1026 14:58:57.443932 (+ 2897us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":253,"tablet-copy.queue_time_us":207639,"thread_start_us":308864,"threads_started":2}
I20251026 14:58:57.767325 23747 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:57.765843 23803 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:57.760345 23798 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.769762 23791 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:57.771299 23677 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:58:57.774199 23755 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:57.726426 22803 raft_consensus.cc:1275] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:57.722275 23633 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:57.466748 23780 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.776520 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:58:57.717772 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 1.04729e+06us
I20251026 14:58:57.747443 23762 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:57.730104 23582 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:57.747117 23740 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:57.711252 23729 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.703522 23796 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.794921 23802 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:58:57.795213 23804 log.cc:927] Time spent T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.294s user 0.001s sys 0.000s
W20251026 14:58:57.798491 23641 consensus_meta.cc:287] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.512s user 0.009s sys 0.000s
I20251026 14:58:57.757885 23661 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:57.717777 23767 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.758529 23698 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:58:57.808308 22697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.020s sys 0.029s
I20251026 14:58:57.812670 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:57.754788 23807 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.813390 23739 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:57.813963 23597 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:57.814365 23694 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:57.818562 23793 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:57.801517 23607 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:57.822969 22801 raft_consensus.cc:2468] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:57.802332 23756 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:57.804513 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:57.775629 23695 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.831322 23763 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:57.831763 23794 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:57.829440 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 908) took 1179 ms. Trace:
I20251026 14:58:57.820883 23766 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:57.817297 23783 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:57.822955 23753 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:57.754937 23809 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.807152 23674 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:57.815878 23699 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:57.883293 22624 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.647s (SENT) [suppressed 31 similar messages]
I20251026 14:58:57.842196 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:57.903944 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 132803us
I20251026 14:58:57.841781 22652 raft_consensus.cc:3060] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:57.841245 22651 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
W20251026 14:58:58.142668 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 445082us
W20251026 14:58:58.148046 22697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.887s user 0.021s sys 0.030s
I20251026 14:58:58.154649 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:58.159121 22800 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:57.827265 23790 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:58:58.161221 22655 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:58.168892 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:58.169209 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:58.169745 22726 raft_consensus.cc:2393] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:58.169867 22730 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:57.829644 23625 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:58.170240 23694 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:58.170863 23763 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.171077 23694 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.172397 23674 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:58.173527 23694 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:57.827714 23787 raft_consensus.cc:493] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:58:58.173785 23813 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:58:58.135010 23805 log.cc:927] Time spent T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.309s user 0.000s sys 0.000s
I20251026 14:58:58.174780 23813 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.175523 23809 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:58.176766 23813 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:57.833978 23789 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:58.142789 22659 rpcz_store.cc:276] 1026 14:58:56.650494 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:56.650706 (+ 212us) service_pool.cc:225] Handling call
1026 14:58:56.867876 (+217170us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:58:56.911949 (+ 44073us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:58:56.912002 (+ 53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:58:57.076817 (+164815us) spinlock_profiling.cc:243] Waited 6.18 ms on lock 0x7b2c00000a78. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cfa5d9b1 00007fd8cfa6cfe2 00007fd8d1199701 00007fd8d115dfda 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22
1026 14:58:57.077352 (+ 535us) log.cc:844] Serialized 26 byte log entry
1026 14:58:57.445246 (+367894us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:58:57.445325 (+ 79us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:58:57.445404 (+ 79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:58:57.820862 (+375458us) raft_consensus.cc:1685] finished
1026 14:58:57.820895 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:58:57.829330 (+ 8435us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43524,"raft.queue_time_us":70391,"raft.run_cpu_time_us":413,"raft.run_wall_time_us":10483,"spinlock_wait_cycles":14214528,"thread_start_us":367417,"threads_started":1,"wal-append.queue_time_us":423030}
I20251026 14:58:58.181175 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:58.188836 22733 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:58.182571 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:58.186601 22622 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:58.186532 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:58.190038 22731 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:57.833720 23784 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:58:58.155426 23791 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:58.196578 23765 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:58.196671 23791 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.197132 22802 raft_consensus.cc:1275] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:58.198777 23791 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:58:58.199829 23746 consensus_meta.cc:287] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.517s user 0.007s sys 0.000s
I20251026 14:58:58.200145 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.200273 23746 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.200810 22659 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:57.931900 23687 tablet_copy_client.cc:806] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Starting download of 0 data blocks...
I20251026 14:58:58.201401 23615 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:58.201774 23687 tablet_copy_client.cc:670] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Starting download of 1 WAL segments...
I20251026 14:58:58.201908 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:58.202351 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:58.202611 23813 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:58.202775 22659 raft_consensus.cc:3060] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:58.203068 22729 raft_consensus.cc:2393] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:58.203109 23813 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.167672 23582 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:58.204030 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:58.204138 23582 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.204659 22729 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:58.202026 23607 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:58.167537 23762 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:58.185161 23597 consensus_queue.cc:1048] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20251026 14:58:58.205235 23813 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:58.136418 23800 raft_consensus.cc:3055] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:58:58.209323 23597 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:58.209596 23800 raft_consensus.cc:740] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:58.209755 23799 consensus_queue.cc:1048] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:58:58.209841 23597 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.211541 22653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32888 (request call id 18) took 1898 ms. Trace:
I20251026 14:58:58.213457 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:58.172983 23763 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:58.225703 22808 raft_consensus.cc:1275] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:58.208024 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:58.230223 23762 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:58.137363 22728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 917) took 1588 ms. Trace:
I20251026 14:58:58.232795 22652 raft_consensus.cc:1275] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:58.145908 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:58:58.244781 22697 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:58:58.221586 23696 consensus_meta.cc:287] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.926s user 0.007s sys 0.004s
I20251026 14:58:58.245743 22731 raft_consensus.cc:2468] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:58.250777 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:58.168779 23790 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:58.212672 22653 rpcz_store.cc:276] 1026 14:58:56.312830 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:56.312981 (+ 151us) service_pool.cc:225] Handling call
1026 14:58:56.313355 (+ 374us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:58:56.466511 (+153156us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:58:56.466547 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:58:57.030911 (+564364us) spinlock_profiling.cc:243] Waited 5.98 ms on lock 0x7b2c00000a78. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cfa5d9b1 00007fd8cfa6cfe2 00007fd8d1199701 00007fd8d115dfda 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22
1026 14:58:57.031491 (+ 580us) log.cc:844] Serialized 26 byte log entry
1026 14:58:57.458423 (+426932us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:58:57.458475 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:58:57.708700 (+250225us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b5400266454. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:58:57.708757 (+ 57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:58:58.211325 (+502568us) raft_consensus.cc:1685] finished
1026 14:58:58.211345 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:58:58.211472 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11890,"raft.queue_time_us":14363,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":99,"spinlock_wait_cycles":337638400,"thread_start_us":426499,"threads_started":1,"wal-append.queue_time_us":746929}
I20251026 14:58:58.251113 22697 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:58.251405 22729 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:58.251941 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.247522 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:58.253100 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:58.174103 23695 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:58.247933 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.199895 22799 raft_consensus.cc:1275] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:58:58.180835 23661 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:58.254859 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.178360 23710 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:58.232503 23739 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:58.250960 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:58.253748 22774 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:58.206140 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:58:58.236649 23582 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:58:58.261363 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 116476us
I20251026 14:58:58.270740 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.281138 22653 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:57.835496 23691 consensus_queue.cc:1048] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.400s
I20251026 14:58:58.210486 23800 consensus_queue.cc:260] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:58.259377 23766 raft_consensus.cc:2804] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:58.211161 23799 consensus_queue.cc:1048] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20251026 14:58:58.207427 23625 consensus_queue.cc:1048] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20251026 14:58:58.275851 22731 raft_consensus.cc:2393] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:58.276042 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:58.275698 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:57.835331 23803 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:58.269954 22805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 555) took 1723 ms. Trace:
I20251026 14:58:58.277159 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:58.279876 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.254137 23809 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:58.247184 23753 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:58.237993 23716 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:58.283517 22651 raft_consensus.cc:1275] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:58.185050 23633 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.259656 23765 raft_consensus.cc:2804] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:58.207283 23607 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.228111 23802 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.166996 23740 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.293608 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.355417 22652 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:58.294440 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:58.303756 23694 raft_consensus.cc:2804] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:58.338634 22733 raft_consensus.cc:2393] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:58.237852 22728 rpcz_store.cc:276] 1026 14:58:56.548744 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:56.549446 (+ 702us) service_pool.cc:225] Handling call
1026 14:58:58.137033 (+1587587us) spinlock_profiling.cc:243] Waited 439 ms on lock 0x7b54002dc654. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:58:58.137226 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1009131776}
I20251026 14:58:58.207046 23784 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:58.274837 23661 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.222313 23787 raft_consensus.cc:515] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:58:58.299086 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 45418.1us
I20251026 14:58:58.227778 23789 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.173557 23793 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:58.330435 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.337803 22805 rpcz_store.cc:276] 1026 14:58:56.546222 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:56.546972 (+ 750us) service_pool.cc:225] Handling call
1026 14:58:56.547352 (+ 380us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:58:56.557923 (+ 10571us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:58:56.557952 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:58:56.563305 (+ 5353us) log.cc:844] Serialized 26 byte log entry
1026 14:58:57.415026 (+851721us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:58:57.415074 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:58:57.415162 (+ 88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:58:58.269657 (+854495us) raft_consensus.cc:1685] finished
1026 14:58:58.269690 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:58:58.269876 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10119,"raft.queue_time_us":15312,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":53845,"thread_start_us":773007,"threads_started":1,"wal-append.queue_time_us":820823}
I20251026 14:58:58.332086 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.333122 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.409623 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:58.336486 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.224538 23794 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.285109 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:58.268034 23790 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:58:58.288089 23402 raft_consensus.cc:3060] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Advancing to term 3
W20251026 14:58:58.339977 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 456170us
W20251026 14:58:58.419776 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:58.261529 23696 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.165779 23641 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.259091 23623 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:58.439333 23623 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:58.440495 23793 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:58.441443 23623 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:58.415616 23698 consensus_queue.cc:1048] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20251026 14:58:58.442816 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:58.347100 23795 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:58.443516 22807 raft_consensus.cc:2468] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:58.357578 22729 raft_consensus.cc:2393] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:58.423194 22803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 542) took 2043 ms. Trace:
I20251026 14:58:58.359692 23633 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:58.360679 22801 raft_consensus.cc:1275] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:58.302994 23582 raft_consensus.cc:493] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:58:58.362937 23742 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:58.346637 23753 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:58.339792 23809 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.201922 23615 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.288393 23739 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.381388 22659 raft_consensus.cc:2468] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
W20251026 14:58:58.381790 23717 consensus_meta.cc:287] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.959s user 0.011s sys 0.000s
I20251026 14:58:58.382715 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.381704 23747 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:58.327832 23812 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:58.333850 23766 raft_consensus.cc:697] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 2 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:58.407863 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.338382 23800 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:58.410409 22805 raft_consensus.cc:2393] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:58.332296 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.409420 22656 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:58.421028 22651 raft_consensus.cc:1240] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 1. Current term is 2. Ops: [1.2-1.2]
I20251026 14:58:58.336936 23767 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:58.346946 23765 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:58.447157 22725 raft_consensus.cc:2393] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:58.383683 23694 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:58.348243 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:58.471567 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:58.472344 22733 raft_consensus.cc:1240] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:58.670533 23694 raft_consensus.cc:3060] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:58.505517 22808 raft_consensus.cc:2368] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 2. Current term is 3.
W20251026 14:58:58.539875 23701 consensus_meta.cc:287] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 1.193s user 0.011s sys 0.000s
W20251026 14:58:58.672814 23684 consensus_meta.cc:287] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.942s user 0.010s sys 0.000s
I20251026 14:58:58.628324 22806 raft_consensus.cc:2468] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
W20251026 14:58:58.553179 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:58.674544 22652 raft_consensus.cc:2468] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:58.677413 22800 raft_consensus.cc:1275] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:58.674937 22622 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:58:58.507225 22553 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38878 (request call id 161) took 3124 ms. Trace:
I20251026 14:58:58.673010 23592 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:58.694414 22774 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:58.695870 22774 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:58.696146 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.703328 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.697539 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.708648 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.701817 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.710968 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.711414 22657 raft_consensus.cc:2468] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
W20251026 14:58:58.540385 23785 consensus_meta.cc:287] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.780s user 0.008s sys 0.003s
I20251026 14:58:58.441512 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:58.714075 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.715960 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:58.716527 22805 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:58.719666 22653 raft_consensus.cc:2410] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 3 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:58:58.721917 22725 raft_consensus.cc:1275] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251026 14:58:58.674227 22655 consensus_meta.cc:287] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.512s user 0.009s sys 0.000s
I20251026 14:58:58.725981 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.726166 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:58.539565 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:58:58.726938 22804 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:58.727187 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:58.726061 23794 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:58.678640 23765 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:58.678097 23797 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:58.729626 22725 raft_consensus.cc:1275] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:58.730178 22733 raft_consensus.cc:1275] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:58.694075 23736 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:58.731251 22725 raft_consensus.cc:1240] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
W20251026 14:58:58.676359 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 208730us
I20251026 14:58:58.694630 23625 raft_consensus.cc:2749] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:58:58.694326 23674 raft_consensus.cc:2764] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20251026 14:58:58.738077 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.720754 23698 raft_consensus.cc:2804] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:58.713301 23686 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:58.696643 23802 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:58.693789 23607 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:58.713150 23633 raft_consensus.cc:2804] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:58.720706 23738 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:58.744089 22659 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:58.748219 23687 tablet_copy_client.cc:538] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 14:58:58.750581 22733 raft_consensus.cc:1275] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:58.751318 23784 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:58.695501 23582 raft_consensus.cc:515] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:58.747179 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:58.722558 23402 raft_consensus.cc:2468] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:58:58.715509 23716 consensus_queue.cc:1048] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20251026 14:58:58.701895 23747 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.731912 22774 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:58.723997 23785 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.740088 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:58.701843 23812 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.756192 22729 raft_consensus.cc:1240] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: []
I20251026 14:58:58.755182 22804 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:58.753690 23686 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.777490 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:58.777999 23747 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:58.778224 22804 raft_consensus.cc:2393] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:58.778366 22733 raft_consensus.cc:1275] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:58:58.778995 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:58.779135 23625 consensus_queue.cc:1048] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:58.779556 22804 raft_consensus.cc:2468] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:58.741484 23797 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.780355 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:58.775889 23802 raft_consensus.cc:2804] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:58.758368 23809 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:58.758540 23615 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:58.754407 23403 raft_consensus.cc:2393] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:58.758611 23746 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:58.759584 23789 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:58.783116 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:58.783778 22807 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:58.784253 23753 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:58.784696 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:58.760681 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:58.785579 22799 raft_consensus.cc:2468] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:58.786520 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:58.788043 22799 raft_consensus.cc:1275] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:58.466328 22803 rpcz_store.cc:276] 1026 14:58:56.380017 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:56.386451 (+ 6434us) service_pool.cc:225] Handling call
1026 14:58:57.138015 (+751564us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b5800002a98. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8ce074451 00007fd8ce0747ce 000000000035a3c9 00007fd8d6091030 00007fd8d6159864 00007fd8d61598d1 00007fd8d611b580 00007fd8d6105ee9 00007fd8d6105992 00007fd8cf69fd34
1026 14:58:57.138377 (+ 362us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:58:58.157319 (+1018942us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:58:58.422895 (+265576us) spinlock_profiling.cc:243] Waited 555 ms on lock 0x7b54003340d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:58:58.423128 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1301297280}
I20251026 14:58:58.790258 23780 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:58.790623 22803 raft_consensus.cc:1275] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:58.791980 22803 raft_consensus.cc:1275] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:58.792080 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:58.793815 23803 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:58.794063 23696 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:58.794996 23597 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:58.762085 23798 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:58.748327 23633 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:58.762936 23729 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:58.748524 22655 raft_consensus.cc:1275] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:58.766467 23739 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:58:58.770720 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:58.721920 23592 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.745728 23698 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:58.771626 22774 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:58.727491 22728 raft_consensus.cc:2468] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:58.745704 23661 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:58.780918 23802 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:58.772153 23607 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:58.752728 23787 leader_election.cc:290] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:58.824864 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:58.761758 23823 ts_tablet_manager.cc:933] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 248493ec312e4398b828c6cb00e5c4b1: Initiating tablet copy from peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:58.681150 22553 rpcz_store.cc:276] 1026 14:58:55.394686 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:55.438958 (+ 44272us) service_pool.cc:225] Handling call
1026 14:58:58.507124 (+3068166us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 14:58:57.033857 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 14:58:57.179981 (+146124us) write_op.cc:432] Acquiring schema lock in shared mode
1026 14:58:57.196600 (+ 16619us) write_op.cc:435] Acquired schema lock
1026 14:58:57.200322 (+ 3722us) tablet.cc:660] Decoding operations
1026 14:58:57.243082 (+ 42760us) write_op.cc:620] Acquiring the partition lock for write op
1026 14:58:57.243175 (+ 93us) write_op.cc:641] Partition lock acquired for write op
1026 14:58:57.243219 (+ 44us) tablet.cc:683] Acquiring locks for 1 operations
1026 14:58:57.243497 (+ 278us) tablet.cc:699] Row locks acquired
1026 14:58:57.243525 (+ 28us) write_op.cc:260] PREPARE: finished
1026 14:58:57.243733 (+ 208us) write_op.cc:270] Start()
1026 14:58:57.243912 (+ 179us) write_op.cc:276] Timestamp: P: 1761490737243668 usec, L: 0
1026 14:58:57.243941 (+ 29us) op_driver.cc:348] REPLICATION: starting
1026 14:58:57.244846 (+ 905us) log.cc:844] Serialized 531 byte log entry
1026 14:58:57.796148 (+551302us) op_driver.cc:464] REPLICATION: finished
1026 14:58:58.468677 (+672529us) write_op.cc:301] APPLY: starting
1026 14:58:58.468787 (+ 110us) tablet.cc:1365] starting BulkCheckPresence
1026 14:58:58.468901 (+ 114us) tablet.cc:1368] finished BulkCheckPresence
1026 14:58:58.469057 (+ 156us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 14:58:58.469329 (+ 272us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 14:58:58.469505 (+ 176us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1026 14:58:58.469551 (+ 46us) write_op.cc:312] APPLY: finished
1026 14:58:58.469749 (+ 198us) log.cc:844] Serialized 22 byte log entry
1026 14:58:58.469930 (+ 181us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:58:58.470010 (+ 80us) write_op.cc:454] Released schema lock
1026 14:58:58.470053 (+ 43us) write_op.cc:341] FINISH: Updating metrics
1026 14:58:58.470855 (+ 802us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:58:58.470886 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":672228,"apply.run_cpu_time_us":1763,"apply.run_wall_time_us":2169,"mutex_wait_us":4847,"num_ops":1,"prepare.queue_time_us":1058671,"prepare.run_cpu_time_us":35216,"prepare.run_wall_time_us":415571,"raft.queue_time_us":831639,"raft.run_cpu_time_us":67516,"raft.run_wall_time_us":920391,"replication_time_us":552029,"thread_start_us":1484176,"threads_started":4,"wal-append.queue_time_us":204470}]]}
I20251026 14:58:58.792093 22800 raft_consensus.cc:1275] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:58.741492 23736 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.813817 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:58.832470 22808 raft_consensus.cc:2468] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:58.799134 22732 raft_consensus.cc:1240] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
I20251026 14:58:58.801520 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.814126 22771 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters:
I20251026 14:58:58.756713 23738 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:58.806139 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:58.796679 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:58.833882 22659 raft_consensus.cc:2468] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 1.
I20251026 14:58:58.834167 23699 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:58.830950 23729 raft_consensus.cc:2804] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:58.813055 23694 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.834785 23740 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:58.834966 23729 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:58.835150 23807 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:58.835465 23729 raft_consensus.cc:3060] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:58.836524 23694 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:58.837534 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:58:58.838276 23592 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:58.838527 22552 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38880 (request call id 169) took 2580 ms. Trace:
I20251026 14:58:58.839478 23765 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.840791 22725 raft_consensus.cc:1240] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: []
I20251026 14:58:58.826717 23698 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:58.812577 23633 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:58.833351 22658 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 2.
I20251026 14:58:58.821488 22733 raft_consensus.cc:1275] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:58.823120 22804 raft_consensus.cc:1275] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:58.824961 22726 raft_consensus.cc:1275] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251026 14:58:58.825310 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 53720.7us
I20251026 14:58:58.784711 23641 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:58.823825 22807 raft_consensus.cc:2368] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 2. Current term is 3.
I20251026 14:58:58.831153 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:58.798983 23796 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:58.838934 22552 rpcz_store.cc:276] 1026 14:58:56.258626 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:56.258820 (+ 194us) service_pool.cc:225] Handling call
1026 14:58:58.838468 (+2579648us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 14:58:57.449584 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 14:58:57.450207 (+ 623us) write_op.cc:432] Acquiring schema lock in shared mode
1026 14:58:57.450263 (+ 56us) write_op.cc:435] Acquired schema lock
1026 14:58:57.450287 (+ 24us) tablet.cc:660] Decoding operations
1026 14:58:57.450651 (+ 364us) write_op.cc:620] Acquiring the partition lock for write op
1026 14:58:57.450705 (+ 54us) write_op.cc:641] Partition lock acquired for write op
1026 14:58:57.450738 (+ 33us) tablet.cc:683] Acquiring locks for 1 operations
1026 14:58:57.450905 (+ 167us) tablet.cc:699] Row locks acquired
1026 14:58:57.450925 (+ 20us) write_op.cc:260] PREPARE: finished
1026 14:58:57.451058 (+ 133us) write_op.cc:270] Start()
1026 14:58:57.451166 (+ 108us) write_op.cc:276] Timestamp: P: 1761490737451015 usec, L: 0
1026 14:58:57.451186 (+ 20us) op_driver.cc:348] REPLICATION: starting
1026 14:58:57.555281 (+104095us) log.cc:844] Serialized 586 byte log entry
1026 14:58:58.322319 (+767038us) op_driver.cc:464] REPLICATION: finished
1026 14:58:58.723547 (+401228us) write_op.cc:301] APPLY: starting
1026 14:58:58.731208 (+ 7661us) tablet.cc:1365] starting BulkCheckPresence
1026 14:58:58.741163 (+ 9955us) tablet.cc:1368] finished BulkCheckPresence
1026 14:58:58.752925 (+ 11762us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 14:58:58.763460 (+ 10535us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 14:58:58.763616 (+ 156us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1026 14:58:58.763663 (+ 47us) write_op.cc:312] APPLY: finished
1026 14:58:58.763856 (+ 193us) log.cc:844] Serialized 22 byte log entry
1026 14:58:58.764025 (+ 169us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:58:58.764114 (+ 89us) write_op.cc:454] Released schema lock
1026 14:58:58.764192 (+ 78us) write_op.cc:341] FINISH: Updating metrics
1026 14:58:58.764591 (+ 399us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:58:58.764621 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":365321,"apply.run_cpu_time_us":1949,"apply.run_wall_time_us":41015,"mutex_wait_us":6800,"num_ops":1,"prepare.queue_time_us":1130688,"prepare.run_cpu_time_us":3211,"prepare.run_wall_time_us":170850,"replication_time_us":844645,"spinlock_wait_cycles":717184,"thread_start_us":380106,"threads_started":1}]]}
I20251026 14:58:58.861804 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:58.826586 23802 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:58.866416 22808 raft_consensus.cc:1275] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:58.866580 22725 raft_consensus.cc:2393] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:58.869635 23370 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:58.870107 23785 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:58.843981 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
W20251026 14:58:58.873358 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 99924.6us
I20251026 14:58:58.876473 22725 raft_consensus.cc:1275] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:58.862154 23597 raft_consensus.cc:2804] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:58:58.879160 22774 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:58.796061 23766 consensus_queue.cc:237] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.897614 22774 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:58:58.828689 23607 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:58.900323 22726 raft_consensus.cc:1275] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:58.902994 22697 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:58.903883 23710 raft_consensus.cc:2804] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:58:58.909118 22697 leader_election.cc:400] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 2. Current term is 3.
I20251026 14:58:58.911211 22697 leader_election.cc:403] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20251026 14:58:58.915966 23823 tablet_copy_client.cc:323] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Beginning tablet copy session from remote peer at address 127.21.2.3:39645
I20251026 14:58:58.916649 23582 leader_election.cc:290] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:58:58.916857 22697 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:58.901749 23641 raft_consensus.cc:2804] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:58.900066 23765 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:58.917723 22697 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:58.913121 22725 raft_consensus.cc:1275] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:58.911921 22726 raft_consensus.cc:1275] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:58.897789 23597 raft_consensus.cc:697] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:58.922195 23807 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:58.894078 23803 consensus_queue.cc:1048] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20251026 14:58:58.893592 23625 consensus_queue.cc:1048] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20251026 14:58:58.907397 23785 raft_consensus.cc:2804] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:58.962771 22733 raft_consensus.cc:1275] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251026 14:58:58.979812 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 96508.8us
I20251026 14:58:58.980855 22801 raft_consensus.cc:1275] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251026 14:58:58.962226 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 44596.2us
I20251026 14:58:58.990350 22697 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:58.922870 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:58.992450 22730 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:58.911125 23710 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:58.922204 23796 raft_consensus.cc:2804] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:58:58.914793 23603 raft_consensus.cc:2804] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20251026 14:58:58.958773 23805 log.cc:927] Time spent T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20251026 14:58:58.988255 23785 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:59.025085 23785 raft_consensus.cc:3060] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
W20251026 14:58:59.030200 23832 log.cc:927] Time spent T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.126s user 0.001s sys 0.000s
I20251026 14:58:59.032711 23687 tablet_bootstrap.cc:492] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap starting.
W20251026 14:58:59.034035 23829 log.cc:927] Time spent T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20251026 14:58:59.038494 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:59.039363 23597 consensus_queue.cc:237] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:58:59.040050 23790 consensus_meta.cc:287] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.557s user 0.002s sys 0.005s
I20251026 14:58:59.040459 23790 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:59.041299 23828 log.cc:927] Time spent T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.160s user 0.001s sys 0.000s
I20251026 14:58:59.041953 22734 raft_consensus.cc:2368] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4.
I20251026 14:58:58.920857 23641 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:58.995731 23740 raft_consensus.cc:2804] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251026 14:58:59.098941 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:59.023509 23710 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:58:59.013931 23603 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:58:59.119562 22622 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:58:59.120246 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 129971us
I20251026 14:58:59.039335 22802 raft_consensus.cc:2468] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
W20251026 14:58:59.010591 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 141047us
I20251026 14:58:59.122025 22730 raft_consensus.cc:1275] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:59.121098 22805 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:58:59.123031 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:59.126014 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.099630 23812 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:59.131161 22800 raft_consensus.cc:2393] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:59.096696 23641 raft_consensus.cc:3060] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:59.098949 22728 raft_consensus.cc:1275] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:59.121434 23603 raft_consensus.cc:3060] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:59.124137 22697 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:59.080569 23698 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:59.133358 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.134733 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:59.140053 23812 raft_consensus.cc:2804] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:59.140349 23686 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:59.140522 23812 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:59.140646 22697 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:59.140959 23812 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:59.141108 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:59.137099 22730 raft_consensus.cc:1275] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:59.142380 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.143395 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.143936 22730 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:59.144464 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:59.144717 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.125706 23796 raft_consensus.cc:697] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 2 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:59.145653 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.145988 23796 consensus_queue.cc:237] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:59.146858 23695 raft_consensus.cc:2804] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251026 14:58:59.125552 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 142899us
I20251026 14:58:59.147986 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.127135 23738 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:59.145174 22729 raft_consensus.cc:2393] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:59.130869 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:59.140391 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:59.285130 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:59.194761 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:59.269708 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:59.130374 23736 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:58:59.253820 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 50842us
I20251026 14:58:59.150410 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:58:59.355412 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:59.147557 23819 raft_consensus.cc:2804] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251026 14:58:59.262259 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 129456us
I20251026 14:58:59.264624 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.130398 23797 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:59.254307 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:59.478906 23698 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:59.261690 23816 consensus_queue.cc:1048] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.235s
I20251026 14:58:59.256145 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:58:59.255121 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 61615us
I20251026 14:58:59.498509 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:59.498219 23785 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:59.499619 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:59.500698 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:59.500831 23785 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:59.288002 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.519477 22659 raft_consensus.cc:2468] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:59.148898 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.520623 22653 raft_consensus.cc:2410] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:58:59.488015 22801 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:59.521446 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.521974 22653 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:59.522603 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:59.522939 22653 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:59.524291 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.525476 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:59.526304 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.526737 22650 raft_consensus.cc:2468] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:59.270211 22806 raft_consensus.cc:2393] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:59.261487 23839 raft_consensus.cc:2804] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:59.294366 23747 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:59.194257 23800 consensus_queue.cc:1048] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.251s
I20251026 14:58:59.287150 23765 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:59.147251 23695 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:59.502662 23716 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:58:59.193950 23696 consensus_queue.cc:1048] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.251s
I20251026 14:58:59.148265 22697 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:58:59.552099 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 403837us
I20251026 14:58:59.517050 23603 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:59.562515 23740 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:59.562975 23740 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:59.564105 23603 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:59.192974 22818 tablet_copy_service.cc:140] P 0d684445287b450a8f738b6ad4b59821: Received BeginTabletCopySession request for tablet 9339edd1ecbf42e3a5e953b2b47f52f2 from peer 248493ec312e4398b828c6cb00e5c4b1 ({username='slave'} at 127.0.0.1:49652)
I20251026 14:58:59.565614 22818 tablet_copy_service.cc:161] P 0d684445287b450a8f738b6ad4b59821: Beginning new tablet copy session on tablet 9339edd1ecbf42e3a5e953b2b47f52f2 from peer 248493ec312e4398b828c6cb00e5c4b1 at {username='slave'} at 127.0.0.1:49652: session id = 248493ec312e4398b828c6cb00e5c4b1-9339edd1ecbf42e3a5e953b2b47f52f2
I20251026 14:58:59.566257 22555 catalog_manager.cc:5649] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 1 to 2, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 2 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:59.567955 22555 catalog_manager.cc:5649] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 1 to 2, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 2 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:59.345320 23790 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:58:59.465742 23843 log.cc:927] Time spent T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
I20251026 14:58:59.196554 23607 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:59.284894 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 501) took 1087 ms. Trace:
I20251026 14:58:59.264273 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:59.255398 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 60576us
I20251026 14:58:59.570578 22732 rpcz_store.cc:276] 1026 14:58:58.181096 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:58.183114 (+ 2018us) service_pool.cc:225] Handling call
1026 14:58:58.183439 (+ 325us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:58:58.833134 (+649695us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:58:59.268139 (+435005us) spinlock_profiling.cc:243] Waited 615 ms on lock 0x7b540040d8d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:58:59.268340 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1415435008}
I20251026 14:58:59.573374 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.573952 22732 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:59.574704 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.574749 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:58:59.477527 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.575759 22657 raft_consensus.cc:2410] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:58:59.576011 23819 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:59.576416 23819 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:59.576588 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.577345 22803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 573) took 1224 ms. Trace:
I20251026 14:58:59.577658 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.578105 22818 tablet_copy_source_session.cc:215] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 14:58:59.578300 22657 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:59.579165 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:59.384833 23802 consensus_meta.cc:287] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.506s user 0.010s sys 0.000s
I20251026 14:58:59.337464 23807 raft_consensus.cc:2749] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 2. Current term is 3.
I20251026 14:58:59.132637 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.581399 23403 raft_consensus.cc:1275] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251026 14:58:59.581188 23802 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:59.581660 22727 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:59.582398 22802 raft_consensus.cc:2393] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:59.582618 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.338791 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.583335 23802 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:59.583767 22728 raft_consensus.cc:2393] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:59.567854 23641 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:59.584687 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.585122 23823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9339edd1ecbf42e3a5e953b2b47f52f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 14:58:59.488301 22658 raft_consensus.cc:2393] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 2: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:58:59.486804 22733 raft_consensus.cc:2376] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2. Re-sending same reply.
I20251026 14:58:59.487284 22651 raft_consensus.cc:2393] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 2: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:58:59.284775 23661 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:59.285059 23784 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:58:59.470381 23841 log.cc:927] Time spent T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.176s user 0.007s sys 0.003s
I20251026 14:58:59.497104 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 505) took 1315 ms. Trace:
I20251026 14:58:59.397910 23812 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:59.329656 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.197389 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:59.502482 22731 raft_consensus.cc:1275] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:59.591104 22726 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:58:59.497182 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:58:59.505470 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:59.507637 23710 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:59.591953 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.592166 23784 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:59.592690 22726 raft_consensus.cc:2393] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 2: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:59.593142 22730 raft_consensus.cc:2393] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:59.593644 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.594007 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.594178 23784 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:59.594597 22730 raft_consensus.cc:2468] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:59.595422 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.589036 23661 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:58:59.508373 22700 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:35092 dropped due to backpressure: service queue is full with 50 items
I20251026 14:58:59.596362 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.510742 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.509837 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:58:59.514003 23633 consensus_meta.cc:287] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.646s user 0.003s sys 0.007s
I20251026 14:58:59.601457 23633 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:59.602607 22800 raft_consensus.cc:2393] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:59.603812 23633 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:59.605438 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:59.520437 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.329046 23809 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:59.606774 23809 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:59.607633 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.608444 22731 raft_consensus.cc:2468] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:59.608718 23809 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:59.315392 23746 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:59.528596 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:59.172508 23370 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:58:59.609437 23746 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:59.610097 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.610232 23836 raft_consensus.cc:2804] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:58:59.610677 22650 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:59.610810 23836 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:59.611321 23836 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:58:59.611747 22727 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:59.612088 22728 raft_consensus.cc:3060] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:59.261750 23699 consensus_queue.cc:1048] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.183s
I20251026 14:58:59.255337 23691 consensus_queue.cc:1048] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.274s
W20251026 14:58:59.613752 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 480449us
I20251026 14:58:59.531045 22624 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:59.615638 23802 consensus_queue.cc:1048] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20251026 14:58:59.616037 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 85122us
I20251026 14:58:59.597326 23710 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:59.616878 22624 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:58:59.285430 23799 consensus_queue.cc:1048] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.243s
I20251026 14:58:59.619024 23799 raft_consensus.cc:2749] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:59.548897 23716 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:59.549198 23845 consensus_queue.cc:1048] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:59.549445 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.519515 23803 consensus_queue.cc:1048] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20251026 14:58:59.639838 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.642550 23716 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:59.644627 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.645663 22727 raft_consensus.cc:2393] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:59.646584 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.650238 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.651373 22731 raft_consensus.cc:2393] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:59.652498 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.549937 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.533102 23625 consensus_queue.cc:1048] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20251026 14:58:59.653901 22659 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:59.545744 23833 consensus_queue.cc:1048] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:58:59.655237 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.528757 23839 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:59.656708 23839 raft_consensus.cc:3060] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
W20251026 14:58:59.531699 23844 log.cc:927] Time spent T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.404s user 0.006s sys 0.002s
I20251026 14:58:59.553004 22697 leader_election.cc:400] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 2. Current term is 3.
I20251026 14:58:59.662348 22697 leader_election.cc:403] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20251026 14:58:59.662694 23716 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:58:59.662935 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 110086us
I20251026 14:58:59.663374 23716 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:59.664788 23784 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 3
I20251026 14:58:59.666126 23716 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:59.665222 23819 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:59.534866 23695 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:59.671456 23819 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:59.669440 23836 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:59.672844 23403 raft_consensus.cc:1240] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20251026 14:58:59.554546 23766 raft_consensus.cc:2804] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:59.673461 23766 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:59.673890 23766 raft_consensus.cc:3060] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:59.674551 23836 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:59.555131 23738 consensus_queue.cc:1048] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:59.556847 23701 consensus_queue.cc:1048] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:58:59.524286 23674 consensus_queue.cc:1048] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20251026 14:58:59.557643 22622 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:58:59.679207 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 121636us
I20251026 14:58:59.533468 23747 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:59.560236 22653 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:58:59.529920 23736 consensus_queue.cc:1048] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20251026 14:58:59.572063 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:59.684432 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 66525.7us
I20251026 14:58:59.684705 22652 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:59.685693 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.686476 22652 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:59.687323 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:59.571940 22555 catalog_manager.cc:5649] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 1 to 2, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 2 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:59.586608 22806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 770) took 1364 ms. Trace:
I20251026 14:58:59.689581 22806 rpcz_store.cc:276] 1026 14:58:58.222065 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:58.824623 (+602558us) service_pool.cc:225] Handling call
1026 14:58:59.586460 (+761837us) spinlock_profiling.cc:243] Waited 443 ms on lock 0x7b540021bbd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:58:59.586565 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1019155712}
I20251026 14:58:59.577730 22803 rpcz_store.cc:276] 1026 14:58:58.346604 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:58.792682 (+446078us) service_pool.cc:225] Handling call
1026 14:58:58.798223 (+ 5541us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:58:58.841346 (+ 43123us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:58:58.841383 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:58:58.841999 (+ 616us) log.cc:844] Serialized 26 byte log entry
1026 14:58:59.027492 (+185493us) spinlock_profiling.cc:243] Waited 3.34 ms on lock 0x7b1800167140. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce0942f7 00007fd8d6e6d219 00007fd8ce0b0c4b 00007fd8ce0aeeef 00007fd8ce0b11b2 00007fd8d0cf82bb 00007fd8d0cfddc3 00007fd8d0cfe171 00007fd8d111965f
1026 14:58:59.256376 (+228884us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:58:59.269228 (+ 12852us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:58:59.296467 (+ 27239us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:58:59.570589 (+274122us) raft_consensus.cc:1685] finished
1026 14:58:59.570619 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:58:59.571051 (+ 432us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2774,"raft.queue_time_us":3827,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":17926,"spinlock_wait_cycles":7676800,"thread_start_us":165047,"threads_started":1,"wal-append.queue_time_us":523458}
I20251026 14:58:59.587586 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.590804 22734 rpcz_store.cc:276] 1026 14:58:58.182001 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:58.207790 (+ 25789us) service_pool.cc:225] Handling call
1026 14:58:59.496927 (+1289137us) spinlock_profiling.cc:243] Waited 811 ms on lock 0x7b5400077954. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:58:59.497049 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1866209536}
I20251026 14:58:59.693471 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.677198 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:59.594425 22805 raft_consensus.cc:2393] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 4: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:58:59.596915 22730 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:58:59.508569 22804 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
W20251026 14:58:59.483839 23729 consensus_meta.cc:287] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.648s user 0.009s sys 0.000s
I20251026 14:58:59.597223 22732 raft_consensus.cc:2468] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:58:59.597707 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.697800 22651 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:59.597834 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.594228 22726 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:59.699302 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:59.699409 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.699919 22730 raft_consensus.cc:2376] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2. Re-sending same reply.
I20251026 14:58:59.698733 22657 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:58:59.701015 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.684118 23625 raft_consensus.cc:2804] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:59.599393 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.600728 23661 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:59.605571 23812 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:59.611375 23746 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:59.534922 23765 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:59.611521 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.533059 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:59.620265 23799 raft_consensus.cc:2749] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:59.621555 22725 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:59.621305 23592 consensus_queue.cc:1059] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20251026 14:58:59.596146 22700 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:35092 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 952)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 968)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 935)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 934)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 966)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 967)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 970)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 960)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 947)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 963)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 961)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 943)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 944)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 936)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 546)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 951)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 547)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 974)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 953)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 956)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 962)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 971)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 548)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 942)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 948)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 552)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 950)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 949)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 938)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 975)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 958)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 930)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 931)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 932)
I20251026 14:58:59.625929 23740 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:59.629627 23641 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:59.635409 22728 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:58:59.693151 23716 consensus_queue.cc:1059] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20251026 14:58:59.727695 23716 raft_consensus.cc:3055] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:58:59.727643 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.728261 23716 raft_consensus.cc:740] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:58:59.728623 22728 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:59.636489 23823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:40908 (request call id 77) took 1430 ms. Trace:
I20251026 14:58:59.729116 23716 consensus_queue.cc:260] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:58:59.730465 23716 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:58:59.729441 23823 rpcz_store.cc:276] 1026 14:58:58.206124 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:58.206258 (+ 134us) service_pool.cc:225] Handling call
1026 14:58:58.835581 (+629323us) ts_tablet_manager.cc:934] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 248493ec312e4398b828c6cb00e5c4b1: Initiating tablet copy from peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
1026 14:58:59.583843 (+748262us) tablet_copy_client.cc:368] Tablet copy session begun
1026 14:58:59.607577 (+ 23734us) tablet_metadata.cc:641] Metadata flushed
1026 14:58:59.607841 (+ 264us) tablet_copy_client.cc:497] Wrote new tablet metadata
1026 14:58:59.635838 (+ 27997us) tablet_copy_client.cc:418] Wrote new consensus metadata
1026 14:58:59.636128 (+ 290us) tablet_replica.cc:206] Creating consensus instance
1026 14:58:59.636368 (+ 240us) ts_tablet_manager.cc:989] Replica registered.
1026 14:58:59.636419 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":14200,"tablet-copy.queue_time_us":547028,"thread_start_us":238264,"threads_started":2}
I20251026 14:58:59.731351 23823 tablet_copy_client.cc:806] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Starting download of 0 data blocks...
I20251026 14:58:59.731948 23823 tablet_copy_client.cc:670] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Starting download of 1 WAL segments...
I20251026 14:58:59.663651 22697 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:58:59.732808 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 69235us
I20251026 14:58:59.195777 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.733937 22655 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:58:59.533478 23837 consensus_queue.cc:1048] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20251026 14:58:59.680543 22729 raft_consensus.cc:2393] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:59.656702 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.683076 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.738124 23746 raft_consensus.cc:2804] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:58:59.690232 22555 catalog_manager.cc:5649] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 2 to 3, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 3 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:58:59.738724 23746 raft_consensus.cc:697] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:58:59.739868 23746 consensus_queue.cc:237] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:59.593032 23607 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:58:59.695003 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:59.694130 22734 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:59.742854 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.743705 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.744362 22734 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:59.745182 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.745759 22734 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:59.697029 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.701894 22726 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:59.747097 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:59.599027 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.748672 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.749405 22733 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:58:59.703426 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.750536 22732 raft_consensus.cc:2368] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3.
I20251026 14:58:59.704398 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.751720 22651 raft_consensus.cc:2393] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 2: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:58:59.711107 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.702633 23729 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:59.725919 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.710246 23625 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:59.743507 23823 tablet_copy_client.cc:538] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 248493ec312e4398b828c6cb00e5c4b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20251026 14:58:59.752122 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 66912.7us
I20251026 14:58:59.752631 22730 raft_consensus.cc:2468] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:59.743542 22729 raft_consensus.cc:2468] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:59.761950 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.763293 22725 raft_consensus.cc:2393] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 2: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:59.744066 23701 consensus_queue.cc:1048] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:58:59.772217 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.773177 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.777088 22734 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:59.777395 22731 raft_consensus.cc:2393] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:59.777194 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.778028 22730 raft_consensus.cc:2468] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:59.777088 22733 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:59.779536 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.779827 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.777982 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.787647 23641 consensus_queue.cc:1048] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:58:59.790551 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.791667 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.792179 22725 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:59.794076 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:59.794839 22806 raft_consensus.cc:2393] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:59.798358 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.798913 22806 raft_consensus.cc:1275] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:59.799110 22732 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:59.780419 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.801371 22730 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:59.764766 22697 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:59.780597 22733 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:59.803342 22733 raft_consensus.cc:1275] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:59.803429 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.804055 22732 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:58:59.804653 22733 raft_consensus.cc:1275] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:59.805780 22730 raft_consensus.cc:1275] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:58:59.807026 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:58:59.713575 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 68107us
W20251026 14:58:59.811558 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 46902.2us
I20251026 14:58:59.811882 22725 raft_consensus.cc:2410] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 3 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:58:59.811836 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:59.777856 23625 raft_consensus.cc:3060] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:59.816792 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:58:59.815014 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:59.820844 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:59.827621 22731 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:59.827977 22652 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:58:59.823534 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:58:59.828028 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 141308us
I20251026 14:58:59.831103 22697 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:58:59.831971 22773 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:59.832983 22773 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:59.824941 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.825841 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:59.826457 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:59.833669 22650 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 1.
I20251026 14:58:59.833920 22654 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:59.833994 22801 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:58:59.836936 23784 raft_consensus.cc:2749] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 2. Current term is 3.
I20251026 14:58:59.821653 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:59.840709 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.840966 22651 raft_consensus.cc:3060] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 3
I20251026 14:58:59.842290 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:58:59.846028 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.847283 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.847975 22733 raft_consensus.cc:2368] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4.
I20251026 14:58:59.787684 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:58:59.858637 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:58:59.742766 23607 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:58:59.850579 23582 raft_consensus.cc:2749] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:59.820211 23661 raft_consensus.cc:2804] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:59.853163 23403 raft_consensus.cc:2393] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 248493ec312e4398b828c6cb00e5c4b1 in this term.
I20251026 14:58:59.854844 23729 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:58:59.854715 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:58:59.877206 22773 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:59.896183 23696 raft_consensus.cc:2804] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:58:59.896683 23696 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:59.897063 23696 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:58:59.848857 22700 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:58:59.866901 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:58:59.863155 23819 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:00.080158 23819 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:58:59.869686 23641 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:58:59.878459 22726 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:00.055989 22654 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
W20251026 14:59:00.078732 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 229958us
I20251026 14:59:00.809859 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:00.810516 22700 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:00.811147 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:00.811802 22654 raft_consensus.cc:3060] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 3
I20251026 14:59:00.811846 22700 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:00.813450 23876 raft_consensus.cc:2804] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:00.813871 23876 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:00.814196 23876 raft_consensus.cc:3060] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:00.813108 22700 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:00.153028 22731 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
W20251026 14:59:00.175650 23864 log.cc:927] Time spent T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.149s user 0.002s sys 0.000s
I20251026 14:59:00.037904 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:00.819469 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:00.820077 22800 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:59:00.033766 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:00.065358 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:00.846357 22801 raft_consensus.cc:2393] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:00.846904 22801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 796) took 1003 ms. Trace:
I20251026 14:59:00.847245 22801 rpcz_store.cc:276] 1026 14:58:59.843678 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:00.065016 (+221338us) service_pool.cc:225] Handling call
1026 14:59:00.846838 (+781822us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:58:59.992918 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:58:59.914219 22799 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
W20251026 14:59:00.300603 22555 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.007s sys 0.029s
I20251026 14:59:00.906159 22799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 794) took 1063 ms. Trace:
W20251026 14:59:00.907100 22555 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.116s user 0.008s sys 0.029s
I20251026 14:59:00.227896 22651 raft_consensus.cc:2468] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:00.225425 22622 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:00.907287 22799 rpcz_store.cc:276] 1026 14:58:59.842196 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:59.854366 (+ 12170us) service_pool.cc:225] Handling call
1026 14:59:00.905859 (+1051493us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:00.836778 22700 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:00.297951 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:00.240567 22652 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:00.917999 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 800) took 1097 ms. Trace:
I20251026 14:59:00.918545 22652 rpcz_store.cc:276] 1026 14:58:59.820457 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:59.820618 (+ 161us) service_pool.cc:225] Handling call
1026 14:59:00.917794 (+1097176us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:00.922961 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 86050.4us
I20251026 14:59:00.927287 22700 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:00.929252 23883 raft_consensus.cc:2749] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:59:00.930374 23883 raft_consensus.cc:2749] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:00.252435 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:00.916847 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:00.940068 22653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 808) took 1092 ms. Trace:
I20251026 14:59:00.828523 22654 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:00.025048 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:58:59.903139 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:01.024570 22808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 797) took 1180 ms. Trace:
I20251026 14:59:01.024855 22808 rpcz_store.cc:276] 1026 14:58:59.844296 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:59.855634 (+ 11338us) service_pool.cc:225] Handling call
1026 14:59:01.024441 (+1168807us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:00.101944 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:59:00.794296 22729 consensus_meta.cc:287] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.981s user 0.019s sys 0.000s
I20251026 14:59:01.026810 22729 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:01.027104 22807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 799) took 1181 ms. Trace:
I20251026 14:59:01.027545 22729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 956) took 1546 ms. Trace:
I20251026 14:59:00.822098 22731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 975) took 1314 ms. Trace:
I20251026 14:58:59.900827 22773 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:01.039290 23691 raft_consensus.cc:2749] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:59.937386 23607 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:00.066870 23625 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:00.034715 23687 tablet_bootstrap.cc:492] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 14:59:00.054972 23695 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.048059 23687 tablet_bootstrap.cc:492] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap complete.
I20251026 14:59:01.049043 22727 raft_consensus.cc:2393] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:01.049787 23687 ts_tablet_manager.cc:1403] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1: Time spent bootstrapping tablet: real 2.017s user 0.037s sys 0.033s
I20251026 14:59:01.050444 23695 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:00.030570 23696 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.053059 22800 raft_consensus.cc:1240] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251026 14:59:01.054322 23696 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:00.292594 23674 raft_consensus.cc:3055] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:59:01.054554 23687 raft_consensus.cc:359] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:59:01.055363 23674 raft_consensus.cc:740] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:01.055788 23687 raft_consensus.cc:740] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Initialized, Role: LEARNER
I20251026 14:59:01.056017 22800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 604) took 1248 ms. Trace:
I20251026 14:59:01.056870 23674 consensus_queue.cc:260] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.056790 23687 consensus_queue.cc:260] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:59:01.057175 22800 rpcz_store.cc:276] 1026 14:58:59.807487 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:00.820637 (+1013150us) service_pool.cc:225] Handling call
1026 14:59:00.820950 (+ 313us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:01.053640 (+232690us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:01.055670 (+ 2030us) spinlock_profiling.cc:243] Waited 231 ms on lock 0x7b54000c6054. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:01.055941 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":532395392}
I20251026 14:59:01.058276 23674 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:59:00.079377 23746 raft_consensus.cc:2749] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:00.000465 23766 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.063628 23766 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:58:59.876564 23784 raft_consensus.cc:2804] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:01.066987 22734 raft_consensus.cc:2393] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:58:59.885599 23799 consensus_queue.cc:1048] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20251026 14:59:01.074038 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 962) took 1591 ms. Trace:
I20251026 14:59:01.074477 22734 rpcz_store.cc:276] 1026 14:58:59.482911 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:59.777758 (+294847us) service_pool.cc:225] Handling call
1026 14:59:01.073754 (+1295996us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b540040d3d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:01.073960 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2933167872}
I20251026 14:59:00.069418 23823 tablet_bootstrap.cc:492] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap starting.
I20251026 14:58:59.995998 23839 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.082837 22656 raft_consensus.cc:2368] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 2. Current term is 3.
I20251026 14:59:01.084070 23839 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:00.780617 23869 log.cc:927] Time spent T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.753s user 0.007s sys 0.000s
I20251026 14:59:00.086690 23819 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:00.827661 23876 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.090025 23607 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:01.090854 23607 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:01.091403 23876 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:00.837262 23878 raft_consensus.cc:2764] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20251026 14:58:59.936580 23402 raft_consensus.cc:1240] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 1. Current term is 2. Ops: []
I20251026 14:59:01.094976 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:00.077984 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:00.095764 23641 raft_consensus.cc:2749] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:58:59.929311 23747 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:00.830091 23812 raft_consensus.cc:2749] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:58:59.891861 23661 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:58:59.903573 23855 consensus_queue.cc:1048] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20251026 14:59:00.786429 22728 raft_consensus.cc:2393] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:00.850255 22805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 605) took 1042 ms. Trace:
I20251026 14:58:59.929966 23765 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:00.248473 23701 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:01.103193 22805 rpcz_store.cc:276] 1026 14:58:59.807715 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:00.843423 (+1035708us) service_pool.cc:225] Handling call
1026 14:59:00.843846 (+ 423us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:00.844230 (+ 384us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:00.844257 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:00.844742 (+ 485us) log.cc:844] Serialized 26 byte log entry
1026 14:59:00.849826 (+ 5084us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:00.849873 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:00.849959 (+ 86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:00.850002 (+ 43us) raft_consensus.cc:1685] finished
1026 14:59:00.850015 (+ 13us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:00.850175 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5672,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":133,"thread_start_us":4651,"threads_started":1,"wal-append.queue_time_us":914}
I20251026 14:59:01.105379 23701 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.106043 22800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 609) took 1280 ms. Trace:
I20251026 14:59:01.106464 22805 raft_consensus.cc:1240] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:01.107175 22805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 802) took 1258 ms. Trace:
I20251026 14:59:01.107642 23701 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:01.106545 22800 rpcz_store.cc:276] 1026 14:58:59.825630 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:01.059075 (+1233445us) service_pool.cc:225] Handling call
1026 14:59:01.059557 (+ 482us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:01.060108 (+ 551us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:01.060137 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:01.061030 (+ 893us) log.cc:844] Serialized 26 byte log entry
1026 14:59:01.070906 (+ 9876us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:01.070955 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:01.071007 (+ 52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:01.105756 (+ 34749us) raft_consensus.cc:1685] finished
1026 14:59:01.105791 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:01.105980 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":400,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":115,"thread_start_us":6024,"threads_started":1,"wal-append.queue_time_us":11036}
I20251026 14:59:00.900520 22655 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.107709 22805 rpcz_store.cc:276] 1026 14:58:59.848561 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:01.105808 (+1257247us) service_pool.cc:225] Handling call
1026 14:59:01.106206 (+ 398us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:01.106935 (+ 729us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:01.107101 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:01.110344 22805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49652 (request call id 60) took 1122 ms. Trace:
I20251026 14:59:01.110605 22805 rpcz_store.cc:276] 1026 14:58:59.987913 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:01.109389 (+1121476us) service_pool.cc:225] Handling call
1026 14:59:01.109874 (+ 485us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:01.110105 (+ 231us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:01.110121 (+ 16us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:01.110163 (+ 42us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:01.110179 (+ 16us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:01.110213 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:01.110286 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:01.112756 22805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49652 (request call id 61) took 1042 ms. Trace:
W20251026 14:59:00.911990 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 686443us
I20251026 14:59:01.113023 22805 rpcz_store.cc:276] 1026 14:59:00.070038 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:01.111884 (+1041846us) service_pool.cc:225] Handling call
1026 14:59:01.112266 (+ 382us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:01.112509 (+ 243us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:01.112524 (+ 15us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:01.112565 (+ 41us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:01.112610 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:01.112645 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:01.112718 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:01.116819 22805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49652 (request call id 62) took 1046 ms. Trace:
W20251026 14:59:00.753973 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 898123us
I20251026 14:59:01.120225 23839 consensus_queue.cc:1048] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:00.910122 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 801) took 1088 ms. Trace:
I20251026 14:59:01.121470 22651 rpcz_store.cc:276] 1026 14:58:59.821262 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:59.821414 (+ 152us) service_pool.cc:225] Handling call
1026 14:59:00.909839 (+1088425us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:01.122573 23837 consensus_queue.cc:1048] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:01.122928 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:00.940701 22653 rpcz_store.cc:276] 1026 14:58:59.847356 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:00.252156 (+404800us) service_pool.cc:225] Handling call
1026 14:59:00.939832 (+687676us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:01.124608 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:01.125624 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:01.126478 22653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 807) took 1286 ms. Trace:
I20251026 14:59:00.920257 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:01.127054 23738 consensus_queue.cc:1059] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:01.127898 22652 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:59:01.128197 23738 consensus_queue.cc:1048] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:00.943508 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:01.130198 22654 raft_consensus.cc:2468] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:59:01.130574 23845 consensus_queue.cc:1048] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:01.131263 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:01.132664 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:01.133301 23845 raft_consensus.cc:3055] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:01.133772 23845 raft_consensus.cc:740] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:00.941013 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:01.135610 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:00.973413 22799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 607) took 1160 ms. Trace:
W20251026 14:59:00.974001 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 47016.7us
W20251026 14:59:00.985567 23802 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.195s user 0.011s sys 0.008s
W20251026 14:59:01.137914 23802 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.348s user 0.012s sys 0.008s
I20251026 14:59:01.138176 22700 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:00.986821 22659 raft_consensus.cc:1240] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251026 14:59:01.139523 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 943) took 1316 ms. Trace:
I20251026 14:59:01.126816 22653 rpcz_store.cc:276] 1026 14:58:59.847073 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:01.125174 (+1278101us) service_pool.cc:225] Handling call
1026 14:59:01.126398 (+ 1224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:01.140025 22659 rpcz_store.cc:276] 1026 14:58:59.823624 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:59.823784 (+ 160us) service_pool.cc:225] Handling call
1026 14:59:00.986152 (+1162368us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:01.139152 (+153000us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:01.139448 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:00.997982 22657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 945) took 1173 ms. Trace:
I20251026 14:59:01.136981 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:01.005568 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:01.141758 22650 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 4
I20251026 14:59:01.141958 23765 raft_consensus.cc:2749] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:01.142525 22803 raft_consensus.cc:2393] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:01.141112 22700 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:01.023131 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 959) took 1540 ms. Trace:
I20251026 14:59:01.028000 22729 rpcz_store.cc:276] 1026 14:58:59.481346 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:59.775590 (+294244us) service_pool.cc:225] Handling call
1026 14:59:01.027451 (+1251861us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:01.031598 23699 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:01.143651 22726 rpcz_store.cc:276] 1026 14:58:59.482147 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:59.746842 (+264695us) service_pool.cc:225] Handling call
1026 14:59:01.022643 (+1275801us) spinlock_profiling.cc:243] Waited 90 ms on lock 0x7b54002dc654. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:01.022841 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":207100160}
I20251026 14:59:01.144367 23699 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.029191 22731 rpcz_store.cc:276] 1026 14:58:59.507678 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:59.794758 (+287080us) service_pool.cc:225] Handling call
1026 14:59:00.822036 (+1027278us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:01.027726 22807 rpcz_store.cc:276] 1026 14:58:59.845785 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:00.101644 (+255859us) service_pool.cc:225] Handling call
1026 14:59:01.026910 (+925266us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:01.145452 23765 raft_consensus.cc:2804] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:01.145956 23765 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:01.141016 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:01.034605 22552 catalog_manager.cc:5649] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 1 to 2, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 2 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:59:01.033658 22555 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38884 (request call id 154) took 4541 ms. Trace:
I20251026 14:59:01.147729 22552 catalog_manager.cc:5649] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 1 to 2, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 2 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:01.037345 22808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 608) took 1220 ms. Trace:
I20251026 14:59:01.149427 22808 rpcz_store.cc:276] 1026 14:58:59.819338 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:01.025508 (+1206170us) service_pool.cc:225] Handling call
1026 14:59:01.025985 (+ 477us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:01.026599 (+ 614us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:01.026638 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:01.027389 (+ 751us) log.cc:844] Serialized 26 byte log entry
1026 14:59:01.037031 (+ 9642us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:01.037058 (+ 27us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:01.037108 (+ 50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:01.037133 (+ 25us) raft_consensus.cc:1685] finished
1026 14:59:01.037140 (+ 7us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:01.037244 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1610,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":172,"thread_start_us":9219,"threads_started":1,"wal-append.queue_time_us":7556}
I20251026 14:59:01.147977 22555 rpcz_store.cc:276] 1026 14:58:56.492193 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:56.492342 (+ 149us) service_pool.cc:225] Handling call
1026 14:59:00.907912 (+4415570us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:00.908007 (+ 95us) write_op.cc:454] Released schema lock
1026 14:59:01.033566 (+125559us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 14:59:00.056565 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 14:59:00.057163 (+ 598us) write_op.cc:432] Acquiring schema lock in shared mode
1026 14:59:00.057204 (+ 41us) write_op.cc:435] Acquired schema lock
1026 14:59:00.057219 (+ 15us) tablet.cc:660] Decoding operations
1026 14:59:00.057875 (+ 656us) write_op.cc:620] Acquiring the partition lock for write op
1026 14:59:00.057932 (+ 57us) write_op.cc:641] Partition lock acquired for write op
1026 14:59:00.057960 (+ 28us) tablet.cc:683] Acquiring locks for 4 operations
1026 14:59:00.059503 (+ 1543us) tablet.cc:699] Row locks acquired
1026 14:59:00.059525 (+ 22us) write_op.cc:260] PREPARE: finished
1026 14:59:00.059691 (+ 166us) write_op.cc:270] Start()
1026 14:59:00.059800 (+ 109us) write_op.cc:276] Timestamp: P: 1761490740059644 usec, L: 0
1026 14:59:00.059820 (+ 20us) op_driver.cc:348] REPLICATION: starting
1026 14:59:00.060614 (+ 794us) log.cc:844] Serialized 1669 byte log entry
1026 14:59:00.113827 (+ 53213us) op_driver.cc:464] REPLICATION: finished
1026 14:59:00.115093 (+ 1266us) write_op.cc:301] APPLY: starting
1026 14:59:00.115180 (+ 87us) tablet.cc:1365] starting BulkCheckPresence
1026 14:59:00.115345 (+ 165us) tablet.cc:1368] finished BulkCheckPresence
1026 14:59:00.115511 (+ 166us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 14:59:00.116098 (+ 587us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 14:59:00.116299 (+ 201us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
1026 14:59:00.116354 (+ 55us) write_op.cc:312] APPLY: finished
1026 14:59:00.116663 (+ 309us) log.cc:844] Serialized 40 byte log entry
1026 14:59:00.117571 (+ 908us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:00.117701 (+ 130us) write_op.cc:454] Released schema lock
1026 14:59:00.117749 (+ 48us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":482011,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":1008,"apply.run_cpu_time_us":2322,"apply.run_wall_time_us":3012,"num_ops":4,"prepare.queue_time_us":260376,"prepare.run_cpu_time_us":12060,"prepare.run_wall_time_us":21589,"raft.queue_time_us":33013,"raft.run_cpu_time_us":1285,"raft.run_wall_time_us":1286,"replication_time_us":53854,"thread_start_us":1160415,"threads_started":4,"wal-append.queue_time_us":17990,"wal-append.run_cpu_time_us":33724,"wal-append.run_wall_time_us":1041667}]]}
I20251026 14:59:01.045090 23625 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:01.151099 22700 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:01.064643 23687 ts_tablet_manager.cc:1434] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1: Time spent starting tablet: real 0.014s user 0.008s sys 0.001s
I20251026 14:59:01.066823 23784 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:01.154366 23784 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.153673 22552 catalog_manager.cc:5649] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 1 to 2, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 2 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:59:01.137458 22799 rpcz_store.cc:276] 1026 14:58:59.813369 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:00.918241 (+1104872us) service_pool.cc:225] Handling call
1026 14:59:00.919955 (+ 1714us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:00.921832 (+ 1877us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:00.921928 (+ 96us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:00.923234 (+ 1306us) log.cc:844] Serialized 26 byte log entry
1026 14:59:00.969764 (+ 46530us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:00.970321 (+ 557us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:00.970460 (+ 139us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:00.973074 (+ 2614us) raft_consensus.cc:1685] finished
1026 14:59:00.973095 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:00.973253 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":398,"raft.run_cpu_time_us":474,"raft.run_wall_time_us":475,"thread_start_us":45668,"threads_started":1,"wal-append.queue_time_us":47175}
I20251026 14:59:01.141453 22657 rpcz_store.cc:276] 1026 14:58:59.824189 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:59.824342 (+ 153us) service_pool.cc:225] Handling call
1026 14:58:59.824646 (+ 304us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:58:59.830316 (+ 5670us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:58:59.830336 (+ 20us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:58:59.855935 (+ 25599us) log.cc:844] Serialized 26 byte log entry
1026 14:59:00.991401 (+1135466us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:00.991540 (+ 139us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:00.991680 (+ 140us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:00.997462 (+ 5782us) raft_consensus.cc:1685] finished
1026 14:59:00.997490 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:00.997734 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5363,"raft.queue_time_us":62509,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":18094,"thread_start_us":6611,"threads_started":1,"wal-append.queue_time_us":7411}
I20251026 14:58:59.977967 23740 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:01.157907 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:59:01.086092 22656 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 768) took 1823 ms (client timeout 2017 ms). Trace:
I20251026 14:59:01.087667 22804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 592) took 1294 ms. Trace:
W20251026 14:59:01.158578 22656 rpcz_store.cc:269] 1026 14:58:59.263158 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:59.264455 (+ 1297us) service_pool.cc:225] Handling call
1026 14:59:01.085781 (+1821326us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b54001fc054. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:01.086008 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3668524800}
I20251026 14:59:01.088057 23696 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:01.164932 22656 raft_consensus.cc:1240] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:01.166028 22656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 809) took 1318 ms. Trace:
I20251026 14:59:01.166325 22656 rpcz_store.cc:276] 1026 14:58:59.847606 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:01.159627 (+1312021us) service_pool.cc:225] Handling call
1026 14:59:01.164704 (+ 5077us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:01.165757 (+ 1053us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:01.165951 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:01.168017 23699 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:01.164932 23696 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.171204 23696 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:01.095885 23402 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40914 (request call id 45) took 1209 ms. Trace:
I20251026 14:59:01.172173 23784 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.172334 23402 rpcz_store.cc:276] 1026 14:58:59.914067 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:59.926452 (+ 12385us) service_pool.cc:225] Handling call
1026 14:58:59.936333 (+ 9881us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:01.095371 (+1159038us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:01.095675 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:01.174436 22657 raft_consensus.cc:2393] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:01.175472 23784 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:01.176182 22657 raft_consensus.cc:1275] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251026 14:59:01.177613 22700 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:01.093168 23607 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:01.178913 23740 raft_consensus.cc:2749] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:01.179847 23740 raft_consensus.cc:2749] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:59:01.117157 22622 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:01.097767 22802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 800) took 1251 ms. Trace:
I20251026 14:59:01.180701 22802 rpcz_store.cc:276] 1026 14:58:59.846381 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:00.077629 (+231248us) service_pool.cc:225] Handling call
1026 14:59:01.097612 (+1019983us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:01.100656 23661 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.103471 22728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 948) took 1733 ms. Trace:
I20251026 14:59:01.181762 22728 rpcz_store.cc:276] 1026 14:58:59.370289 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:59.787369 (+417080us) service_pool.cc:225] Handling call
1026 14:59:01.103228 (+1315859us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:00.855783 22801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 606) took 1042 ms. Trace:
I20251026 14:59:01.183009 22651 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.182961 22801 rpcz_store.cc:276] 1026 14:58:59.813072 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:00.847965 (+1034893us) service_pool.cc:225] Handling call
1026 14:59:00.848338 (+ 373us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:00.850709 (+ 2371us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:00.850732 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:00.851167 (+ 435us) log.cc:844] Serialized 26 byte log entry
1026 14:59:00.852311 (+ 1144us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:00.852352 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:00.852605 (+ 253us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:00.855377 (+ 2772us) raft_consensus.cc:1685] finished
1026 14:59:00.855407 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:00.855697 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1476,"raft.run_cpu_time_us":57,"raft.run_wall_time_us":58,"thread_start_us":2719,"threads_started":2,"wal-append.queue_time_us":1109}
I20251026 14:59:01.129027 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:01.143043 22803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44594 (request call id 593) took 1349 ms. Trace:
I20251026 14:59:01.185428 22803 rpcz_store.cc:276] 1026 14:58:59.793362 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:59.797941 (+ 4579us) service_pool.cc:225] Handling call
1026 14:59:01.005030 (+1207089us) spinlock_profiling.cc:243] Waited 6.71 ms on lock 0x7b5800002a98. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8ce074451 00007fd8ce074302 00007fd8d6f0dad9 00007fd8d6090d30 00007fd8d615984b 00007fd8d61598d1 00007fd8d6107660 00007fd8d6106f44 00007fd8cf6a4834 00007fd8cf6a47c2
1026 14:59:01.142978 (+137948us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15422208}
I20251026 14:59:01.015384 22697 leader_election.cc:304] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
W20251026 14:59:01.187390 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 172413us
I20251026 14:59:01.187474 23845 consensus_queue.cc:260] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.020779 22806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 803) took 1170 ms. Trace:
I20251026 14:59:01.146358 23765 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:01.189414 22806 rpcz_store.cc:276] 1026 14:58:59.849846 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:00.012215 (+162369us) service_pool.cc:225] Handling call
1026 14:59:01.020518 (+1008303us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:01.189831 23855 raft_consensus.cc:2804] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:01.190316 23855 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:01.190682 23855 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.191535 23740 raft_consensus.cc:2749] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:01.192350 22652 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.152631 22653 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 3
W20251026 14:59:01.039899 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 1.13888e+06us
I20251026 14:59:01.194743 22773 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:01.195927 22773 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:01.196759 22806 raft_consensus.cc:1275] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:59:01.044023 22658 raft_consensus.cc:1240] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:01.198529 22658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 802) took 1376 ms. Trace:
I20251026 14:59:01.198823 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:01.198982 22658 rpcz_store.cc:276] 1026 14:58:59.822076 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:59.822206 (+ 130us) service_pool.cc:225] Handling call
1026 14:58:59.822418 (+ 212us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:01.197606 (+1375188us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:01.198250 (+ 644us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400265554. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:01.198467 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2808180608}
W20251026 14:59:01.200359 22755 debug-util.cc:398] Leaking SignalData structure 0x7b080026d2c0 after lost signal to thread 21515
W20251026 14:59:01.051400 22727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 935) took 1684 ms (client timeout 1914 ms). Trace:
W20251026 14:59:01.201719 22755 debug-util.cc:398] Leaking SignalData structure 0x7b0800263540 after lost signal to thread 22590
W20251026 14:59:01.201402 22727 rpcz_store.cc:269] 1026 14:58:59.366461 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:59.646282 (+279821us) service_pool.cc:225] Handling call
1026 14:59:01.051011 (+1404729us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54000ba4d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:01.051312 (+ 301us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3221482752}
W20251026 14:59:01.202369 22755 debug-util.cc:398] Leaking SignalData structure 0x7b08002d6ae0 after lost signal to thread 22604
I20251026 14:59:01.203007 23625 consensus_queue.cc:1048] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251026 14:59:01.203855 22755 debug-util.cc:398] Leaking SignalData structure 0x7b0800167fa0 after lost signal to thread 22683
I20251026 14:59:01.146358 22654 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.158648 22804 rpcz_store.cc:276] 1026 14:58:59.792898 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:59.793039 (+ 141us) service_pool.cc:225] Handling call
1026 14:58:59.793371 (+ 332us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:58:59.802770 (+ 9399us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:58:59.802799 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:58:59.826843 (+ 24044us) log.cc:844] Serialized 26 byte log entry
1026 14:58:59.979536 (+152693us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:58:59.979599 (+ 63us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:58:59.979698 (+ 99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:00.780070 (+800372us) spinlock_profiling.cc:243] Waited 6.84 ms on lock 0x7b440002304c. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8cdfc0905 00007fd8cdfc1322 00007fd8d0d0a231 00007fd8d0d05b5a 00007fd8d0cf991e 00007fd8d0cf8c66 00007fd8d0cf8888 00007fd8d0d022c1 00007fd8d0d02279 00007fd8d0d02209
1026 14:59:01.087032 (+306962us) raft_consensus.cc:1685] finished
1026 14:59:01.087078 (+ 46us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:01.087403 (+ 325us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":88,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":6600,"spinlock_wait_cycles":15738240,"thread_start_us":123951,"threads_started":1,"wal-append.queue_time_us":148889}
I20251026 14:59:01.193427 23701 consensus_queue.cc:1059] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20251026 14:59:01.207307 23857 consensus_queue.cc:1048] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20251026 14:59:01.204711 23625 consensus_queue.cc:1048] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20251026 14:59:01.205256 22755 debug-util.cc:398] Leaking SignalData structure 0x7b08002a9e60 after lost signal to thread 22758
W20251026 14:59:01.212401 22755 debug-util.cc:398] Leaking SignalData structure 0x7b080010d340 after lost signal to thread 22759
W20251026 14:59:01.213661 22755 debug-util.cc:398] Leaking SignalData structure 0x7b08002f69e0 after lost signal to thread 22832
W20251026 14:59:01.214144 22755 debug-util.cc:398] Leaking SignalData structure 0x7b0800287440 after lost signal to thread 22833
I20251026 14:59:01.207814 22773 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:01.215948 22755 debug-util.cc:398] Leaking SignalData structure 0x7b0800346840 after lost signal to thread 23428
W20251026 14:59:01.182199 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 65226.4us
I20251026 14:59:01.217458 23699 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.208688 23607 raft_consensus.cc:2804] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:01.228440 23607 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:01.228586 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:01.228904 23607 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.211314 23701 raft_consensus.cc:3055] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:01.228986 22800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49652 (request call id 63) took 1053 ms. Trace:
I20251026 14:59:01.253763 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:01.255545 22652 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:01.255779 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:01.256160 23701 raft_consensus.cc:740] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:59:01.238024 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 51284.9us
I20251026 14:59:01.257017 22800 rpcz_store.cc:276] 1026 14:59:00.175432 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:01.136340 (+960908us) service_pool.cc:225] Handling call
1026 14:59:01.136970 (+ 630us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:01.137252 (+ 282us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:01.137300 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:01.228410 (+ 91110us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:01.228454 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:01.228531 (+ 77us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:01.228913 (+ 382us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:01.257740 23701 consensus_queue.cc:260] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.259140 23701 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.259972 23765 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.237675 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:01.261826 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 84432.4us
I20251026 14:59:01.235859 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:01.262641 22700 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:01.238705 22801 raft_consensus.cc:1275] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:59:01.240559 23696 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.251569 22773 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:01.117238 22805 rpcz_store.cc:276] 1026 14:59:00.070335 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:01.115701 (+1045366us) service_pool.cc:225] Handling call
1026 14:59:01.116150 (+ 449us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:01.116485 (+ 335us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:01.116512 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:01.116563 (+ 51us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:01.116600 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:01.116661 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:01.116768 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:01.270557 23641 raft_consensus.cc:2804] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:01.271049 23641 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:01.271420 23641 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.239579 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:01.263883 23740 raft_consensus.cc:2804] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:01.273552 23740 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:01.273737 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 56563.2us
I20251026 14:59:01.273964 23740 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.236160 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:01.189169 23845 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.286551 22654 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:01.287130 22654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 949) took 1067 ms. Trace:
I20251026 14:59:01.287438 22654 rpcz_store.cc:276] 1026 14:59:00.219885 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:01.132310 (+912425us) service_pool.cc:225] Handling call
1026 14:59:01.287073 (+154763us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:01.226513 23661 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:59:01.290083 23661 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:01.219658 22697 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:01.297819 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:01.292887 22650 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:01.308677 22650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 959) took 1086 ms. Trace:
I20251026 14:59:01.301920 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:01.309131 22650 rpcz_store.cc:276] 1026 14:59:00.222585 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:01.136659 (+914074us) service_pool.cc:225] Handling call
1026 14:59:01.308607 (+171948us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:01.256143 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 960) took 1033 ms. Trace:
I20251026 14:59:01.310655 22652 rpcz_store.cc:276] 1026 14:59:00.222975 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:01.128667 (+905692us) service_pool.cc:225] Handling call
1026 14:59:01.256065 (+127398us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:01.312722 22653 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:01.313443 22653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 957) took 1091 ms. Trace:
I20251026 14:59:01.313823 22653 rpcz_store.cc:276] 1026 14:59:00.222023 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:01.140591 (+918568us) service_pool.cc:225] Handling call
1026 14:59:01.313373 (+172782us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:01.289036 22654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 969) took 1064 ms. Trace:
W20251026 14:59:01.317409 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 54850.2us
I20251026 14:59:01.317620 22654 rpcz_store.cc:276] 1026 14:59:00.224713 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:01.288086 (+1063373us) service_pool.cc:225] Handling call
1026 14:59:01.288461 (+ 375us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:01.288724 (+ 263us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:01.288749 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:01.288792 (+ 43us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:01.288821 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:01.288871 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:01.288969 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:01.326023 23738 consensus_queue.cc:1048] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20251026 14:59:01.328616 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 71327.4us
W20251026 14:59:01.333766 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 109709us
I20251026 14:59:01.340516 22651 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:01.341403 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 958) took 1119 ms. Trace:
I20251026 14:59:01.341812 22651 rpcz_store.cc:276] 1026 14:59:00.222293 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:01.124158 (+901865us) service_pool.cc:225] Handling call
1026 14:59:01.341324 (+217166us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:01.343844 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 124330us
I20251026 14:59:01.356792 23699 raft_consensus.cc:2749] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:01.376588 23765 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:01.378793 22658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 967) took 1149 ms. Trace:
I20251026 14:59:01.378649 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 961) took 1151 ms. Trace:
W20251026 14:59:01.346607 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 109438us
I20251026 14:59:01.378188 23833 consensus_queue.cc:1059] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20251026 14:59:01.380686 22773 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:01.392925 22700 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:01.379271 22658 rpcz_store.cc:276] 1026 14:59:00.224313 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:01.200203 (+975890us) service_pool.cc:225] Handling call
1026 14:59:01.200671 (+ 468us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:01.205426 (+ 4755us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:01.205456 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:01.206010 (+ 554us) log.cc:844] Serialized 26 byte log entry
1026 14:59:01.324052 (+118042us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:01.324090 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:01.324171 (+ 81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:01.373941 (+ 49770us) raft_consensus.cc:1685] finished
1026 14:59:01.373964 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:01.374104 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7689,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":129,"thread_start_us":117562,"threads_started":1,"wal-append.queue_time_us":117087}
W20251026 14:59:01.384606 23892 log.cc:927] Time spent T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.099s user 0.001s sys 0.000s
I20251026 14:59:01.381063 23784 raft_consensus.cc:2804] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:01.398147 22657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 966) took 1174 ms. Trace:
I20251026 14:59:01.400187 22657 rpcz_store.cc:276] 1026 14:59:00.224118 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:01.177235 (+953117us) service_pool.cc:225] Handling call
1026 14:59:01.177675 (+ 440us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:01.178009 (+ 334us) raft_consensus.cc:1534] Early marking committed up to index 2
1026 14:59:01.305201 (+127192us) log.cc:844] Serialized 14 byte log entry
1026 14:59:01.394548 (+ 89347us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:01.394597 (+ 49us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:01.397813 (+ 3216us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:01.397915 (+ 102us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:01.398076 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12699,"prepare.queue_time_us":120105,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"thread_start_us":30014,"threads_started":1}
I20251026 14:59:01.387982 23696 raft_consensus.cc:2749] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251026 14:59:01.381577 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 87602.7us
I20251026 14:59:01.379505 22659 rpcz_store.cc:276] 1026 14:59:00.223169 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:01.168719 (+945550us) service_pool.cc:225] Handling call
1026 14:59:01.169171 (+ 452us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:01.169505 (+ 334us) raft_consensus.cc:1534] Early marking committed up to index 2
1026 14:59:01.169857 (+ 352us) log.cc:844] Serialized 14 byte log entry
1026 14:59:01.374333 (+204476us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:01.374442 (+ 109us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:01.374509 (+ 67us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:01.374603 (+ 94us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:01.374777 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":167,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":23179,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":131,"thread_start_us":200115,"threads_started":2,"wal-append.queue_time_us":129405}
I20251026 14:59:01.410033 23899 consensus_queue.cc:1048] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20251026 14:59:01.410375 22697 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:01.411916 23661 raft_consensus.cc:2804] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:01.412346 23661 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:01.412706 23661 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.413026 23765 raft_consensus.cc:2804] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:01.413575 23765 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:01.413956 23765 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.414319 23740 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.415759 23699 raft_consensus.cc:2749] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:01.417562 22656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 952) took 1197 ms. Trace:
I20251026 14:59:01.417757 23740 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:01.417963 22656 rpcz_store.cc:276] 1026 14:59:00.220496 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:01.167017 (+946521us) service_pool.cc:225] Handling call
1026 14:59:01.167435 (+ 418us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:01.167868 (+ 433us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:01.167899 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:01.168412 (+ 513us) log.cc:844] Serialized 26 byte log entry
1026 14:59:01.289497 (+121085us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:01.289533 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:01.289593 (+ 60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:01.417341 (+127748us) raft_consensus.cc:1685] finished
1026 14:59:01.417360 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:01.417501 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4288,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100,"thread_start_us":120710,"threads_started":1,"wal-append.queue_time_us":116809}
I20251026 14:59:01.402136 23784 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:01.439612 23812 consensus_queue.cc:1059] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20251026 14:59:01.440092 23905 consensus_queue.cc:1048] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:01.441077 23812 raft_consensus.cc:3055] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:01.441521 23812 raft_consensus.cc:740] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:01.442564 23812 consensus_queue.cc:260] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.444393 23812 raft_consensus.cc:3060] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.430512 23784 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
W20251026 14:59:01.451332 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 94298.5us
I20251026 14:59:01.452500 22773 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:01.453861 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 106514us
I20251026 14:59:01.459175 23699 raft_consensus.cc:2804] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:59:01.464443 23699 raft_consensus.cc:697] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:01.465983 23699 consensus_queue.cc:237] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.438516 23641 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.468135 22806 raft_consensus.cc:2393] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
W20251026 14:59:01.498986 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 54070.6us
W20251026 14:59:01.542207 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 87339.4us
I20251026 14:59:01.429563 23897 consensus_queue.cc:1048] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.126s
W20251026 14:59:01.551993 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 155646us
I20251026 14:59:01.435627 23845 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:01.472000 23641 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:01.566145 22773 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:01.527293 23855 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.571920 23661 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.585558 23696 raft_consensus.cc:2804] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:01.583026 22773 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:01.607112 22700 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:01.607600 22806 raft_consensus.cc:1240] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:01.608163 23740 raft_consensus.cc:2764] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20251026 14:59:01.609354 22774 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:01.610152 22624 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:01.610932 23802 raft_consensus.cc:2804] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:01.611038 23897 consensus_queue.cc:1048] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:01.611608 23915 raft_consensus.cc:2749] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:01.611503 23802 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:01.612262 23802 raft_consensus.cc:3060] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.612192 22624 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:01.612550 22700 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:01.614491 23915 raft_consensus.cc:3060] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:59:01.616044 23740 raft_consensus.cc:2749] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:01.610733 22773 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:01.619158 22773 leader_election.cc:304] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:01.627822 23823 tablet_bootstrap.cc:492] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251026 14:59:01.637686 23661 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:01.612655 23803 raft_consensus.cc:2749] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:01.598876 22655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 944) took 1774 ms. Trace:
W20251026 14:59:01.640568 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:01.642634 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:01.606006 23845 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.656179 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:01.657127 22657 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:59:01.612134 23819 consensus_queue.cc:1048] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
W20251026 14:59:01.668715 21517 debug-util.cc:398] Leaking SignalData structure 0x7b080000fe20 after lost signal to thread 22773
I20251026 14:59:01.671140 23661 consensus_queue.cc:1048] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:01.642397 22655 rpcz_store.cc:276] 1026 14:58:59.823955 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:59.826399 (+ 2444us) service_pool.cc:225] Handling call
1026 14:59:00.899970 (+1073571us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:01.554947 (+654977us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:01.558040 (+ 3093us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:01.567462 (+ 9422us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:01.572449 (+ 4987us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:01.576960 (+ 4511us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:01.593912 (+ 16952us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":73100,"raft.queue_time_us":42785,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":14999,"thread_start_us":70426,"threads_started":1}
I20251026 14:59:01.673748 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:01.674899 23904 raft_consensus.cc:3055] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:01.675452 23904 raft_consensus.cc:740] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:01.660422 22700 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:01.676406 23904 consensus_queue.cc:260] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.677610 22700 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:01.678529 22656 raft_consensus.cc:2376] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3. Re-sending same reply.
I20251026 14:59:01.681108 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:01.681972 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:01.694265 22803 raft_consensus.cc:2393] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
W20251026 14:59:01.682736 23908 log.cc:927] Time spent T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.129s user 0.002s sys 0.000s
I20251026 14:59:01.700320 22624 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:01.683704 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:01.689564 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:01.671332 23915 raft_consensus.cc:2749] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:59:01.692598 22804 raft_consensus.cc:3060] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.653867 23641 raft_consensus.cc:2804] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:01.720166 23904 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.722846 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:01.660398 23823 tablet_bootstrap.cc:492] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 248493ec312e4398b828c6cb00e5c4b1: Bootstrap complete.
I20251026 14:59:01.727550 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:59:01.729470 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 119096us
I20251026 14:59:01.735080 22657 raft_consensus.cc:2376] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3. Re-sending same reply.
I20251026 14:59:01.723708 23661 raft_consensus.cc:2749] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:01.664099 23607 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.727403 23641 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:01.765777 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:01.773484 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:59:01.776074 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 157002us
I20251026 14:59:01.776166 23916 consensus_queue.cc:1048] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.085s
W20251026 14:59:01.777568 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:01.734545 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 127749us
W20251026 14:59:01.781713 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 118953us
I20251026 14:59:01.787734 22659 raft_consensus.cc:2468] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:59:01.782099 23855 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:01.782521 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 104977us
I20251026 14:59:01.746470 23857 raft_consensus.cc:2749] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:59:01.777882 23917 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:01.778224 23740 raft_consensus.cc:2804] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:01.781035 23925 raft_consensus.cc:2749] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:01.806349 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:01.779299 22744 tablet_copy_service.cc:342] P 562bb1e1c8e445b8baecd29c3669658e: Request end of tablet copy session 248493ec312e4398b828c6cb00e5c4b1-2efb26bccebf43b5b8a85f51fd977714 received from {username='slave'} at 127.0.0.1:47996
I20251026 14:59:01.787160 22800 raft_consensus.cc:1240] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251026 14:59:01.807286 22655 raft_consensus.cc:2393] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:01.807512 22744 tablet_copy_service.cc:434] P 562bb1e1c8e445b8baecd29c3669658e: ending tablet copy session 248493ec312e4398b828c6cb00e5c4b1-2efb26bccebf43b5b8a85f51fd977714 on tablet 2efb26bccebf43b5b8a85f51fd977714 with peer 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:01.807632 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:01.808506 22651 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:01.774327 22657 raft_consensus.cc:2376] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3. Re-sending same reply.
I20251026 14:59:01.791110 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:01.768299 23823 ts_tablet_manager.cc:1403] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 248493ec312e4398b828c6cb00e5c4b1: Time spent bootstrapping tablet: real 1.699s user 0.046s sys 0.007s
I20251026 14:59:01.790416 22773 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:01.769544 23641 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.795037 22774 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:01.789474 23929 raft_consensus.cc:2804] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:01.797827 23802 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.803939 23784 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.804220 22624 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:01.805491 23857 consensus_queue.cc:1059] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:59:01.770237 23696 raft_consensus.cc:697] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:01.810864 22804 raft_consensus.cc:2468] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:01.813105 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:01.794935 23740 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:01.822057 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:01.823037 22659 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:01.818697 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:01.814272 23929 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:01.826676 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:01.828732 22654 raft_consensus.cc:2393] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:01.829319 23925 raft_consensus.cc:2804] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:59:01.831281 23925 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:01.831862 23925 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:59:01.831498 23823 raft_consensus.cc:359] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.835565 23607 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:01.838389 23916 raft_consensus.cc:2804] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:01.842567 23916 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:01.843222 23916 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.824785 23929 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.843058 22700 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:01.842397 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:01.846750 22654 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
W20251026 14:59:01.851017 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 62771.5us
I20251026 14:59:01.825137 23740 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.845292 23765 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
W20251026 14:59:01.854816 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 50825.7us
I20251026 14:59:01.834236 23923 raft_consensus.cc:2804] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251026 14:59:01.856016 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:01.857579 23923 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:01.859480 23823 raft_consensus.cc:740] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Initialized, Role: LEARNER
I20251026 14:59:01.859653 23923 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:01.869043 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:01.857023 23845 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:01.885949 22624 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:01.875092 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 53293.7us
I20251026 14:59:01.865756 22773 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:01.873413 23927 consensus_queue.cc:1059] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20251026 14:59:01.828578 23917 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:01.860409 23812 raft_consensus.cc:2749] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251026 14:59:01.891327 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 48740.6us
I20251026 14:59:01.905160 22700 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:01.916289 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:01.866912 23607 consensus_queue.cc:1048] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20251026 14:59:01.888803 23625 raft_consensus.cc:2749] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:59:01.894712 23819 consensus_queue.cc:1059] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
W20251026 14:59:01.922979 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 56753.1us
I20251026 14:59:01.923192 23765 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:59:01.923475 23819 raft_consensus.cc:3055] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:59:01.923925 23819 raft_consensus.cc:740] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:01.927515 23917 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:01.917483 23823 consensus_queue.cc:260] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:59:01.932067 23823 ts_tablet_manager.cc:1434] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 248493ec312e4398b828c6cb00e5c4b1: Time spent starting tablet: real 0.119s user 0.007s sys 0.002s
I20251026 14:59:01.934135 22700 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:01.936141 22700 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:01.923848 22624 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
W20251026 14:59:01.939347 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 46175.1us
I20251026 14:59:01.925081 23819 consensus_queue.cc:260] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:01.975178 22552 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38880 (request call id 170) took 1769 ms. Trace:
I20251026 14:59:01.972997 23784 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:01.936937 23812 raft_consensus.cc:2749] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:59:01.940858 23802 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:01.957794 23845 raft_consensus.cc:2804] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:59:01.984921 22622 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:01.986095 22774 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:02.031041 22818 tablet_copy_service.cc:342] P 0d684445287b450a8f738b6ad4b59821: Request end of tablet copy session 248493ec312e4398b828c6cb00e5c4b1-9339edd1ecbf42e3a5e953b2b47f52f2 received from {username='slave'} at 127.0.0.1:49652
I20251026 14:59:02.031641 22818 tablet_copy_service.cc:434] P 0d684445287b450a8f738b6ad4b59821: ending tablet copy session 248493ec312e4398b828c6cb00e5c4b1-9339edd1ecbf42e3a5e953b2b47f52f2 on tablet 9339edd1ecbf42e3a5e953b2b47f52f2 with peer 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:01.982203 23915 raft_consensus.cc:3055] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:59:01.919922 23696 consensus_queue.cc:237] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:01.967620 23661 raft_consensus.cc:2804] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251026 14:59:02.037890 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 179516us
I20251026 14:59:02.002530 22799 raft_consensus.cc:1240] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:02.067785 22654 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:01.978199 22552 rpcz_store.cc:276] 1026 14:59:00.198566 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:00.790909 (+592343us) service_pool.cc:225] Handling call
1026 14:59:01.967329 (+1176420us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 14:59:01.280266 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 14:59:01.280904 (+ 638us) write_op.cc:432] Acquiring schema lock in shared mode
1026 14:59:01.280958 (+ 54us) write_op.cc:435] Acquired schema lock
1026 14:59:01.280975 (+ 17us) tablet.cc:660] Decoding operations
1026 14:59:01.281705 (+ 730us) write_op.cc:620] Acquiring the partition lock for write op
1026 14:59:01.281850 (+ 145us) write_op.cc:641] Partition lock acquired for write op
1026 14:59:01.281892 (+ 42us) tablet.cc:683] Acquiring locks for 3 operations
1026 14:59:01.282428 (+ 536us) tablet.cc:699] Row locks acquired
1026 14:59:01.282456 (+ 28us) write_op.cc:260] PREPARE: finished
1026 14:59:01.282677 (+ 221us) write_op.cc:270] Start()
1026 14:59:01.282886 (+ 209us) write_op.cc:276] Timestamp: P: 1761490741282598 usec, L: 0
1026 14:59:01.282931 (+ 45us) op_driver.cc:348] REPLICATION: starting
1026 14:59:01.283887 (+ 956us) log.cc:844] Serialized 1289 byte log entry
1026 14:59:01.571665 (+287778us) op_driver.cc:464] REPLICATION: finished
1026 14:59:01.695324 (+123659us) spinlock_profiling.cc:243] Waited 7.68 ms on lock 0x7b440002304c. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8cdfc0905 00007fd8cdfc1322 00007fd8d0d0a231 00007fd8d0d05b5a 00007fd8ce09459c 00007fd8d6e6d219 00007fd8ce0b0c4b 00007fd8ce0aeeef 00007fd8ce0b11b2 00007fd8d4b8c650
1026 14:59:01.851601 (+156277us) write_op.cc:301] APPLY: starting
1026 14:59:01.853416 (+ 1815us) tablet.cc:1365] starting BulkCheckPresence
1026 14:59:01.875619 (+ 22203us) tablet.cc:1368] finished BulkCheckPresence
1026 14:59:01.890688 (+ 15069us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 14:59:01.933863 (+ 43175us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 14:59:01.934134 (+ 271us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
1026 14:59:01.934200 (+ 66us) write_op.cc:312] APPLY: finished
1026 14:59:01.934463 (+ 263us) log.cc:844] Serialized 34 byte log entry
1026 14:59:01.934646 (+ 183us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:01.934741 (+ 95us) write_op.cc:454] Released schema lock
1026 14:59:01.934811 (+ 70us) write_op.cc:341] FINISH: Updating metrics
1026 14:59:01.935246 (+ 435us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:01.935277 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":50812,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":243898,"apply.run_cpu_time_us":2829,"apply.run_wall_time_us":98361,"num_ops":3,"prepare.queue_time_us":105003,"prepare.run_cpu_time_us":5462,"prepare.run_wall_time_us":40372,"raft.queue_time_us":95827,"raft.run_cpu_time_us":25101,"raft.run_wall_time_us":271111,"replication_time_us":241740,"spinlock_wait_cycles":17659648,"thread_start_us":367636,"threads_started":4,"wal-append.queue_time_us":35083}]]}
I20251026 14:59:02.011801 23784 raft_consensus.cc:2749] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251026 14:59:02.086618 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:02.098574 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:02.011790 23674 raft_consensus.cc:2804] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:02.039705 23845 raft_consensus.cc:697] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:59:02.107613 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 183852us
I20251026 14:59:02.112310 22624 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:02.105023 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:02.097843 23661 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:02.038918 23915 raft_consensus.cc:740] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:59:02.107254 23888 log.cc:927] Time spent T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20251026 14:59:02.125525 23661 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:02.138177 23607 raft_consensus.cc:2804] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:02.138691 23607 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:02.139110 23607 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:02.140287 23950 raft_consensus.cc:2804] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:02.141973 23934 consensus_queue.cc:1059] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.151s
I20251026 14:59:02.145138 22624 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:02.146319 23839 consensus_queue.cc:1059] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:02.148356 23740 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.149390 23944 raft_consensus.cc:3055] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:02.149809 23944 raft_consensus.cc:740] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:02.150732 23944 consensus_queue.cc:260] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.152122 23944 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
W20251026 14:59:02.138343 23935 log.cc:927] Time spent T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.196s user 0.004s sys 0.000s
I20251026 14:59:02.164242 22658 raft_consensus.cc:2410] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:02.168159 22624 leader_election.cc:400] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507) with higher term. Message: Invalid argument: T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3.
I20251026 14:59:02.169999 22624 leader_election.cc:403] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20251026 14:59:02.171016 23833 raft_consensus.cc:2749] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507) with higher term. Message: Invalid argument: T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3.
I20251026 14:59:02.171447 23923 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:59:02.172736 22624 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:02.168964 23839 raft_consensus.cc:3055] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:02.141044 23950 raft_consensus.cc:697] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:02.169413 23934 raft_consensus.cc:3055] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:02.168664 23912 raft_consensus.cc:2749] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 14:59:02.174255 22773 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:02.179245 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:02.155102 23857 raft_consensus.cc:2764] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20251026 14:59:02.137616 23674 raft_consensus.cc:697] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:02.175879 23839 raft_consensus.cc:740] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:02.200465 23951 raft_consensus.cc:2804] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:59:02.198277 22553 catalog_manager.cc:5649] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:59:02.207300 23951 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:02.207993 23951 raft_consensus.cc:3060] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:59:02.187125 23934 raft_consensus.cc:740] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:02.202711 23845 consensus_queue.cc:237] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.215577 23950 consensus_queue.cc:237] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:02.217648 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 84595.9us
I20251026 14:59:02.193919 23904 raft_consensus.cc:2764] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20251026 14:59:02.218717 22773 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:02.220000 23699 raft_consensus.cc:2804] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:02.220629 23699 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:02.221194 23699 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
W20251026 14:59:02.225584 23898 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.668s user 0.016s sys 0.022s
W20251026 14:59:02.226092 23898 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.668s user 0.016s sys 0.022s
I20251026 14:59:02.220177 23925 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.232429 23934 consensus_queue.cc:260] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.233928 23934 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
W20251026 14:59:02.239815 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 67142.9us
I20251026 14:59:02.214398 23923 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:02.212361 23929 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:02.251305 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 293638us
I20251026 14:59:02.256110 22624 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:02.236740 23641 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.259016 22804 raft_consensus.cc:1240] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:02.261472 23929 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:02.261996 23925 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:02.262511 23819 raft_consensus.cc:3060] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:59:02.187523 23915 consensus_queue.cc:260] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:02.274395 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 338293us
I20251026 14:59:02.281227 22806 raft_consensus.cc:1240] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:02.283669 22700 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:02.293476 22773 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:02.296247 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:02.295852 23661 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.287370 22774 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:02.298681 23661 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:02.273088 23904 raft_consensus.cc:2764] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20251026 14:59:02.273945 23740 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:02.246023 23839 consensus_queue.cc:260] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:02.312559 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 67161.9us
I20251026 14:59:02.305310 23696 raft_consensus.cc:2804] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:02.313860 23696 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:02.314306 23696 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:02.314764 23641 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:02.309993 22774 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:02.317859 22624 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:02.310223 23959 raft_consensus.cc:2749] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:02.306090 23916 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.322618 23898 raft_consensus.cc:2749] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:02.319813 23925 raft_consensus.cc:2804] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:02.327250 23661 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:02.323732 23740 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:02.331995 23661 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.332342 23740 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.334220 23661 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:02.335085 23740 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:02.323206 23839 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:02.338419 22624 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:02.344300 23898 raft_consensus.cc:2804] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:59:02.344820 23898 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:02.345245 23898 raft_consensus.cc:3060] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:59:02.344939 22700 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:02.346263 23740 raft_consensus.cc:2749] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:59:02.347751 22700 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:02.348944 23740 raft_consensus.cc:2749] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:02.337157 22774 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:02.352905 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:02.341907 23925 raft_consensus.cc:697] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:02.360242 23925 consensus_queue.cc:237] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.362159 22773 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:02.364059 23699 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.368654 23857 consensus_queue.cc:1059] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:59:02.338308 23641 raft_consensus.cc:2804] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:02.339989 23915 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:59:02.383796 23941 raft_consensus.cc:2804] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:02.398586 23941 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:02.399065 23941 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:02.402542 23699 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:02.402904 23929 consensus_queue.cc:1048] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:02.389529 23857 consensus_queue.cc:1059] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.008s
I20251026 14:59:02.414287 23857 raft_consensus.cc:3055] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:02.414757 23857 raft_consensus.cc:740] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:02.416038 23402 raft_consensus.cc:1275] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251026 14:59:02.391369 23904 raft_consensus.cc:3055] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:02.417449 23904 raft_consensus.cc:740] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:02.418399 23904 consensus_queue.cc:260] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.419552 23904 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:02.420374 23929 consensus_queue.cc:1048] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 14:59:02.415639 23857 consensus_queue.cc:260] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.423098 23857 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:02.389619 23641 raft_consensus.cc:697] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:02.424947 23641 consensus_queue.cc:237] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.426501 23916 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:02.438119 22773 connection.cc:232] RPC call timeout handler was delayed by -1.6807077930000105s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21530000000000002s for a response
W20251026 14:59:02.438927 22773 connection.cc:232] RPC call timeout handler was delayed by -1.6549704110000221s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2721s for a response
W20251026 14:59:02.439348 22773 connection.cc:232] RPC call timeout handler was delayed by -1.61860779299991s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2222s for a response
W20251026 14:59:02.441350 22773 connection.cc:232] RPC call timeout handler was delayed by -1.5943077929999845s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22490000000000002s for a response
W20251026 14:59:02.441777 22773 connection.cc:232] RPC call timeout handler was delayed by -1.5250833399999237s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2819s for a response
I20251026 14:59:02.440063 23951 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:02.445115 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 113478us
I20251026 14:59:02.445394 23944 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:02.447397 23944 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:02.324064 23674 consensus_queue.cc:237] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.451380 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:02.452220 23403 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:02.453948 22771 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters:
I20251026 14:59:02.455034 23929 raft_consensus.cc:2804] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:02.455420 23929 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:02.455754 23929 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:02.461390 23944 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:02.465019 23929 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:02.466265 22773 connection.cc:232] RPC call timeout handler was delayed by -1.3970901879998792s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1516s for a response
W20251026 14:59:02.468312 22773 connection.cc:232] RPC call timeout handler was delayed by -1.3598031759999003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15710000000000002s for a response
W20251026 14:59:02.468811 22773 connection.cc:232] RPC call timeout handler was delayed by -1.1013822319999917s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.166s for a response
W20251026 14:59:02.469174 22773 connection.cc:232] RPC call timeout handler was delayed by -1.0113822309999705s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20070000000000002s for a response
I20251026 14:59:02.471760 23740 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:02.472397 23740 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.474817 23740 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:02.475351 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:02.476084 23403 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:02.501118 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 153459us
I20251026 14:59:02.502197 22700 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:02.504802 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 167779us
I20251026 14:59:02.492453 23929 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:02.445184 23951 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:02.508950 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:02.509112 22700 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:02.512205 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:02.510947 23699 raft_consensus.cc:2804] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:02.510847 23740 raft_consensus.cc:2749] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:02.508090 23696 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.521452 22700 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:02.517102 23661 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:02.525619 22806 raft_consensus.cc:1240] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:02.525527 23661 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.519472 23857 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:02.526479 23857 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:02.527328 23699 raft_consensus.cc:697] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:02.527598 23661 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:02.528168 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:02.516371 23607 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.528282 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:02.536971 22774 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:02.528399 23699 consensus_queue.cc:237] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.528872 23857 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:02.548851 22802 raft_consensus.cc:1240] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
I20251026 14:59:02.550369 22799 raft_consensus.cc:1240] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:02.556689 22800 raft_consensus.cc:1240] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:02.557857 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:02.517041 23819 raft_consensus.cc:2749] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:02.558920 23857 consensus_queue.cc:1059] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:59:02.559934 23857 raft_consensus.cc:3055] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:02.560322 23857 raft_consensus.cc:740] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:59:02.530512 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:02.561175 23857 consensus_queue.cc:260] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.563094 23857 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:02.562395 22555 catalog_manager.cc:5649] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: term changed from 1 to 3, leader changed from df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 3 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:59:02.566793 23941 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.558032 23607 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:02.549731 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:02.556653 23403 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:02.576781 22658 raft_consensus.cc:2468] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 2.
W20251026 14:59:02.580817 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:02.584971 22800 raft_consensus.cc:1240] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
I20251026 14:59:02.599073 23950 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:59:02.602612 23916 consensus_queue.cc:1059] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:59:02.603899 23916 raft_consensus.cc:3055] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:02.604277 23916 raft_consensus.cc:740] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:02.605096 23916 consensus_queue.cc:260] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.609313 23916 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:02.569227 23941 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:02.621570 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 84677.5us
I20251026 14:59:02.622505 22771 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters:
W20251026 14:59:02.621493 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 100106us
I20251026 14:59:02.570812 23696 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:02.623646 22700 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:02.596168 23819 raft_consensus.cc:2749] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:02.607308 23845 consensus_queue.cc:1059] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20251026 14:59:02.627312 22552 catalog_manager.cc:5649] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 1 to 2, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 2 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:02.633975 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:02.628504 22622 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:02.634557 22656 raft_consensus.cc:2376] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 2. Re-sending same reply.
I20251026 14:59:02.635690 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:02.584745 23898 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:02.637892 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 82255.1us
I20251026 14:59:02.626114 23699 raft_consensus.cc:2804] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:02.641145 22774 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:02.639616 22552 catalog_manager.cc:5649] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 1 to 2, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 2 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:02.632249 23740 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:02.646628 22552 catalog_manager.cc:5649] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 1 to 2, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 2 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:02.650202 22700 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:02.652817 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:02.637351 23934 consensus_queue.cc:1059] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20251026 14:59:02.644179 23674 raft_consensus.cc:2749] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:02.650839 22552 catalog_manager.cc:5649] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 1 to 2, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 2 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:02.657305 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:02.657889 22552 catalog_manager.cc:5649] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 1 to 2, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 2 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:02.658495 22774 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:02.666440 23951 raft_consensus.cc:2804] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:02.667534 23370 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:02.666989 22552 catalog_manager.cc:5649] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 1 to 2, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 2 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:02.669035 23917 raft_consensus.cc:2804] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:59:02.665167 23929 raft_consensus.cc:2804] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:02.670797 23929 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:02.671195 23929 raft_consensus.cc:3060] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:02.654145 23699 raft_consensus.cc:697] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:02.663827 22774 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:02.674485 23699 consensus_queue.cc:237] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:02.675966 23641 raft_consensus.cc:2804] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:02.676376 23641 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:02.676791 23641 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:02.646745 23833 consensus_queue.cc:1059] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20251026 14:59:02.656879 23740 raft_consensus.cc:697] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:02.678755 23915 raft_consensus.cc:3055] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:59:02.678484 23740 consensus_queue.cc:237] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.679145 23915 raft_consensus.cc:740] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:02.679636 22774 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:02.679971 23915 consensus_queue.cc:260] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.680863 23941 raft_consensus.cc:2804] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:02.681504 23941 raft_consensus.cc:697] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:02.692394 22658 raft_consensus.cc:2468] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 3.
I20251026 14:59:02.669394 22552 catalog_manager.cc:5649] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 1 to 2, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 2 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
W20251026 14:59:02.693109 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 45189.3us
I20251026 14:59:02.697042 22774 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:02.685154 23819 raft_consensus.cc:2804] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:02.667033 23951 raft_consensus.cc:697] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:02.700464 22552 catalog_manager.cc:5649] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 1 to 2, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 2 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:02.713390 22700 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:02.711297 22552 catalog_manager.cc:5649] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 2 to 3, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 3 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:02.721189 22774 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:02.710335 23925 raft_consensus.cc:2749] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:59:02.730224 22774 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:02.715444 23845 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:02.715876 23915 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:59:02.716415 23819 raft_consensus.cc:697] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:02.726538 23917 raft_consensus.cc:697] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 LEADER]: Becoming Leader. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:02.735428 22552 catalog_manager.cc:5649] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 1 to 2, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 2 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:02.748939 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:02.732404 22803 raft_consensus.cc:1240] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:02.750113 22656 raft_consensus.cc:2410] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:02.750305 22552 catalog_manager.cc:5649] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 1 to 2, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 2 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:59:02.753841 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:02.756314 22658 raft_consensus.cc:3060] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
W20251026 14:59:02.756639 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 50371.6us
I20251026 14:59:02.735976 23607 raft_consensus.cc:2804] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:02.740305 23950 raft_consensus.cc:2749] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:59:02.760720 22552 catalog_manager.cc:5649] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 1 to 2, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 2 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:02.743723 23944 raft_consensus.cc:2804] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election won for term 3
W20251026 14:59:02.759305 22624 leader_election.cc:336] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 2.946s (SENT)
I20251026 14:59:02.742123 23765 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:02.764145 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:02.771026 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:02.769121 22624 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:02.769903 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:02.749255 22799 raft_consensus.cc:1240] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:02.773752 22800 raft_consensus.cc:2368] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3.
I20251026 14:59:02.768985 22552 catalog_manager.cc:5649] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 1 to 3, leader changed from df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 3 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:59:02.775153 22552 catalog_manager.cc:5649] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 1 to 2, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 2 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:02.777200 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:02.740516 23845 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.778470 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:02.779184 23845 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:02.760121 23607 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:02.779553 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:02.779811 23607 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:02.780602 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:02.780862 23765 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.781687 23917 consensus_queue.cc:237] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:02.782975 23765 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:02.783572 23944 raft_consensus.cc:697] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:02.784593 23944 consensus_queue.cc:237] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.784988 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:02.739472 23370 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:02.786586 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 61787.8us
I20251026 14:59:02.787853 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:02.756705 23661 raft_consensus.cc:2749] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:02.789112 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:02.742321 23959 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:02.791328 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:02.793000 22654 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:02.794121 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:02.794245 22658 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:02.796902 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:02.797524 22800 raft_consensus.cc:2468] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:59:02.798434 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:02.798972 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:02.799640 22803 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:02.800238 22655 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:02.801374 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:02.801263 22552 catalog_manager.cc:5649] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 1 to 2, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 2 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:02.802992 22552 catalog_manager.cc:5649] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 1 to 2, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 2 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } }
I20251026 14:59:02.804438 22552 catalog_manager.cc:5649] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 1 to 2, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 2 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:59:02.806221 22700 leader_election.cc:400] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 2. Current term is 3.
I20251026 14:59:02.769690 23898 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:02.820640 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:59:02.823904 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:02.823895 22552 catalog_manager.cc:5649] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 1 to 2, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 2 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:02.829497 22622 leader_election.cc:400] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3.
I20251026 14:59:02.829973 22622 leader_election.cc:403] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20251026 14:59:02.805897 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:02.830982 22622 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:02.843317 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:02.844874 22806 raft_consensus.cc:1240] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251026 14:59:02.823987 23845 raft_consensus.cc:2749] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:02.753404 23951 consensus_queue.cc:237] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.769112 23941 consensus_queue.cc:237] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:02.856289 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 97022.8us
I20251026 14:59:02.833720 23899 raft_consensus.cc:2749] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3.
I20251026 14:59:02.861687 23904 raft_consensus.cc:3055] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:59:02.827769 23916 consensus_queue.cc:1059] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
W20251026 14:59:02.871233 23971 log.cc:927] Time spent T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.164s user 0.004s sys 0.000s
I20251026 14:59:02.843075 23934 raft_consensus.cc:2804] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251026 14:59:02.872383 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 80026.7us
I20251026 14:59:02.862643 23912 raft_consensus.cc:3055] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:02.883590 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:02.831563 23833 consensus_queue.cc:1059] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20251026 14:59:02.916851 23833 consensus_queue.cc:1243] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer 0d684445287b450a8f738b6ad4b59821 log is divergent from this leader: its last log entry 1.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251026 14:59:02.897902 23899 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:02.917528 23833 raft_consensus.cc:3055] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:02.917903 23833 raft_consensus.cc:740] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:02.917867 23899 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:02.918830 23976 log.cc:927] Time spent T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.088s user 0.000s sys 0.001s
I20251026 14:59:02.920084 23899 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:02.920711 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:02.892230 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:02.833910 23959 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.882736 23904 raft_consensus.cc:740] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:02.942932 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:02.943535 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:02.944145 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:02.945303 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:02.945883 22801 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:59:02.895455 23980 raft_consensus.cc:2749] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251026 14:59:02.901747 23934 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:02.946648 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:02.946986 23934 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:02.947526 22727 raft_consensus.cc:2393] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:02.913340 23740 consensus_queue.cc:1059] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:02.948200 23740 raft_consensus.cc:3055] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:02.948549 23740 raft_consensus.cc:740] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:02.949355 23740 consensus_queue.cc:260] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.949769 23959 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:02.915472 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:02.902771 23912 raft_consensus.cc:740] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:02.911938 23911 raft_consensus.cc:3055] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Stepping down as leader of term 2
W20251026 14:59:02.897459 22624 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 3.222s (SENT) [suppressed 53 similar messages]
I20251026 14:59:02.950855 22726 raft_consensus.cc:2393] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:02.953941 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:02.954427 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:02.954792 22734 raft_consensus.cc:2410] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:02.955221 22733 raft_consensus.cc:2393] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:02.955662 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:02.955936 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:02.956396 22734 raft_consensus.cc:2393] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
W20251026 14:59:02.879436 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 92107.4us
I20251026 14:59:02.957074 23370 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:02.957664 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:02.957862 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:02.956498 22733 raft_consensus.cc:2376] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3. Re-sending same reply.
I20251026 14:59:02.959121 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:02.957124 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:02.959812 22730 raft_consensus.cc:2468] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:59:02.960220 22725 raft_consensus.cc:2468] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:59:02.960685 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:02.961251 23978 log.cc:927] Time spent T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20251026 14:59:02.961818 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:02.962447 22732 raft_consensus.cc:2468] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:59:02.961145 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:02.963551 22725 raft_consensus.cc:2393] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:02.964509 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:02.965106 22725 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:02.965540 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:02.966073 22727 raft_consensus.cc:1240] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251026 14:59:02.849661 23819 consensus_queue.cc:237] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:02.957231 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:02.920814 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:02.969987 22806 raft_consensus.cc:2393] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:02.944348 22807 raft_consensus.cc:2393] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:02.946667 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:02.963413 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:02.969700 22734 raft_consensus.cc:2468] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
W20251026 14:59:02.984488 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:03.008728 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:02.910533 22651 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:03.040635 23950 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:02.935839 23899 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:03.028489 22730 raft_consensus.cc:2393] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:02.951256 23911 raft_consensus.cc:740] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:59:03.012607 23967 log.cc:927] Time spent T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1: Append to log took a long time: real 0.089s user 0.004s sys 0.000s
W20251026 14:59:03.169750 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:03.172783 22725 raft_consensus.cc:2468] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
W20251026 14:59:03.093600 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 196148us
I20251026 14:59:03.072433 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:03.156062 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 149912us
W20251026 14:59:03.152042 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 51058.5us
W20251026 14:59:03.071552 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:03.174948 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:02.918833 23833 consensus_queue.cc:260] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:03.073171 23661 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:03.176256 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:03.176745 22731 raft_consensus.cc:2393] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 4: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:03.179718 22732 raft_consensus.cc:2393] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:03.156630 23959 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:03.183377 23980 raft_consensus.cc:2804] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:03.182204 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:03.197132 22654 raft_consensus.cc:1240] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: []
W20251026 14:59:03.197480 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:03.185779 22651 raft_consensus.cc:1275] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:59:03.184222 22774 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:02.986006 23912 consensus_queue.cc:260] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:03.225821 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:03.226861 22730 raft_consensus.cc:2393] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:03.253234 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:03.254101 22731 raft_consensus.cc:2393] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:03.184696 23661 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:03.166484 23607 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:03.176543 23899 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:03.171208 23950 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:03.248077 23977 log.cc:927] Time spent T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.346s user 0.000s sys 0.001s
I20251026 14:59:03.158787 23904 consensus_queue.cc:260] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:03.241467 23833 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:03.259400 23696 raft_consensus.cc:2804] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:03.302440 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:03.302798 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:03.206511 23959 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:03.306229 22734 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
W20251026 14:59:03.307060 23986 log.cc:927] Time spent T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20251026 14:59:03.307792 23959 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:03.308346 23959 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:03.309896 23981 log.cc:927] Time spent T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20251026 14:59:03.310003 23696 raft_consensus.cc:697] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:03.311156 23696 consensus_queue.cc:237] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:03.308826 23959 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:03.329879 22804 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:03.314232 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:03.297860 23912 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:03.353474 23899 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:03.261560 23911 consensus_queue.cc:260] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:03.355005 23857 consensus_meta.cc:287] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.791s user 0.010s sys 0.001s
I20251026 14:59:03.356104 23607 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:03.357012 23950 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:03.358759 23980 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:03.359210 23980 raft_consensus.cc:3060] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Advancing to term 4
W20251026 14:59:03.360174 23641 consensus_meta.cc:287] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.683s user 0.008s sys 0.000s
I20251026 14:59:03.360601 23641 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:03.361629 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:03.362349 22731 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
W20251026 14:59:03.366473 23988 log.cc:927] Time spent T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.212s user 0.001s sys 0.000s
W20251026 14:59:03.367882 23929 consensus_meta.cc:287] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.696s user 0.005s sys 0.004s
I20251026 14:59:03.368261 23929 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:03.329154 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 50404.8us
W20251026 14:59:02.950261 23765 consensus_queue.cc:1175] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0d684445287b450a8f738b6ad4b59821" responder_term: 3 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 2. Current term is 3. Ops: [2.2-2.2]" } } last_received_current_leader { term: 0 index: 0 } }
I20251026 14:59:03.374730 22729 raft_consensus.cc:1240] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20251026 14:59:03.321596 22803 consensus_meta.cc:287] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.521s user 0.009s sys 0.001s
I20251026 14:59:03.397084 22732 raft_consensus.cc:3060] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:03.398262 22734 raft_consensus.cc:2468] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
W20251026 14:59:03.402832 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:03.403427 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:03.424438 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:03.414989 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:03.437090 22800 raft_consensus.cc:1240] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20251026 14:59:03.401175 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 103648us
I20251026 14:59:03.520464 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:03.378335 23929 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
W20251026 14:59:03.480146 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:03.595371 22734 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
W20251026 14:59:03.464910 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:03.609191 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:03.609982 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:03.611460 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:03.612139 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:03.615475 22657 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
W20251026 14:59:03.616201 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:03.436311 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 64481.2us
W20251026 14:59:03.610122 23901 log.cc:927] Time spent T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d: Append to log took a long time: real 0.161s user 0.001s sys 0.000s
I20251026 14:59:03.616784 23765 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:03.622275 23765 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:03.624681 23765 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:03.625351 22725 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:03.625667 22801 raft_consensus.cc:1275] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:59:03.628019 23998 consensus_queue.cc:1048] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:03.632367 23661 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:03.635706 23904 raft_consensus.cc:3060] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:59:03.609987 22805 raft_consensus.cc:2468] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:03.639034 22622 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:03.643173 23641 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:03.644476 23980 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:03.584976 23951 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:03.645556 23951 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:03.647469 23951 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:03.649683 22651 raft_consensus.cc:1275] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251026 14:59:03.652928 23403 raft_consensus.cc:1275] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251026 14:59:03.601716 23999 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:03.441869 22807 raft_consensus.cc:1240] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:03.653730 23999 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:03.654378 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:03.369246 23934 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:03.655845 23999 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:03.656610 23934 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:03.656402 23999 raft_consensus.cc:2804] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:03.657325 23999 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:03.657646 23999 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:03.658784 22799 raft_consensus.cc:1240] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
W20251026 14:59:03.661976 23915 consensus_meta.cc:287] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.916s user 0.012s sys 0.000s
I20251026 14:59:03.666002 23959 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:03.670543 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:03.671990 23999 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:03.672427 22553 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38880 (request call id 171) took 1516 ms. Trace:
I20251026 14:59:03.656394 23985 consensus_queue.cc:1059] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251026 14:59:03.676017 23985 raft_consensus.cc:3055] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:03.676537 23985 raft_consensus.cc:740] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:03.678352 23699 consensus_queue.cc:1048] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20251026 14:59:03.689565 22622 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:03.691442 23898 raft_consensus.cc:2804] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:03.692008 23898 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:03.692451 23898 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:03.693929 23985 consensus_queue.cc:260] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:03.694649 23999 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:03.695300 23985 raft_consensus.cc:3060] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Advancing to term 3
W20251026 14:59:03.699806 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:03.678452 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:03.706188 22806 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:03.707516 22622 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:03.712399 22734 raft_consensus.cc:2468] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:59:03.700702 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:03.714529 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:03.714057 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 529890us
I20251026 14:59:03.723834 22732 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:03.716766 22774 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:03.725975 22731 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:03.726702 22774 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:03.727137 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:03.727936 22731 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
W20251026 14:59:03.700459 22650 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.503s user 0.014s sys 0.020s
I20251026 14:59:03.729944 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:03.849684 22656 raft_consensus.cc:2468] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:59:03.930197 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:03.742401 22729 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:03.731900 22725 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
W20251026 14:59:03.732133 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:03.955473 22732 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:03.856647 22657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1005) took 1080 ms. Trace:
I20251026 14:59:03.952316 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:03.959563 22725 raft_consensus.cc:2393] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
W20251026 14:59:03.967535 22773 leader_election.cc:336] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.563s (SENT)
I20251026 14:59:03.968329 22773 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:03.969997 22734 raft_consensus.cc:3060] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:59:03.957208 22657 rpcz_store.cc:276] 1026 14:59:02.776023 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:02.780386 (+ 4363us) service_pool.cc:225] Handling call
1026 14:59:02.820402 (+ 40016us) spinlock_profiling.cc:243] Waited 9.1 ms on lock 0x7b5800000998. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8ce074451 00007fd8ce074302 00007fd8d6f0dad9 00007fd8d6090d30 00007fd8d615984b 00007fd8d61598d1 00007fd8d6107660 00007fd8d6106f44 00007fd8cf6a4834 00007fd8cf6a47c2
1026 14:59:03.856386 (+1035984us) spinlock_profiling.cc:243] Waited 536 ms on lock 0x7b540034d354. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:03.856555 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1253558784}
W20251026 14:59:03.973834 22699 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.676s user 0.018s sys 0.057s
W20251026 14:59:03.974227 22699 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.768s user 0.018s sys 0.058s
I20251026 14:59:03.975576 23994 consensus_queue.cc:1048] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.158s
I20251026 14:59:03.770480 23765 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:03.977659 23765 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:03.978713 23929 raft_consensus.cc:2749] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251026 14:59:03.949177 23368 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.007s sys 0.004s
I20251026 14:59:03.401629 22803 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:03.742925 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:59:03.995733 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:03.761186 23944 raft_consensus.cc:2804] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:03.756692 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:04.007473 22804 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:03.942487 23403 raft_consensus.cc:1240] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
W20251026 14:59:03.912417 22727 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.592s user 0.023s sys 0.030s
I20251026 14:59:03.989781 22650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1001) took 1218 ms. Trace:
I20251026 14:59:03.739741 23999 raft_consensus.cc:2804] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:04.009876 22729 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:04.011426 22808 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:03.855690 23641 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:04.014499 23765 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:04.003300 23368 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.579s user 0.008s sys 0.004s
I20251026 14:59:04.015273 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:04.018329 24010 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:04.021229 22731 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
W20251026 14:59:03.774121 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:03.757845 23699 raft_consensus.cc:2804] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:04.007879 22774 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:03.716042 23959 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:03.672834 22553 rpcz_store.cc:276] 1026 14:59:02.155742 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:02.160688 (+ 4946us) service_pool.cc:225] Handling call
1026 14:59:03.672367 (+1511679us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 14:59:02.338222 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 14:59:02.475187 (+136965us) write_op.cc:432] Acquiring schema lock in shared mode
1026 14:59:02.475225 (+ 38us) write_op.cc:435] Acquired schema lock
1026 14:59:02.475244 (+ 19us) tablet.cc:660] Decoding operations
1026 14:59:02.475574 (+ 330us) write_op.cc:620] Acquiring the partition lock for write op
1026 14:59:02.475629 (+ 55us) write_op.cc:641] Partition lock acquired for write op
1026 14:59:02.475658 (+ 29us) tablet.cc:683] Acquiring locks for 1 operations
1026 14:59:02.475840 (+ 182us) tablet.cc:699] Row locks acquired
1026 14:59:02.475859 (+ 19us) write_op.cc:260] PREPARE: finished
1026 14:59:02.475995 (+ 136us) write_op.cc:270] Start()
1026 14:59:02.476099 (+ 104us) write_op.cc:276] Timestamp: P: 1761490742475950 usec, L: 0
1026 14:59:02.476120 (+ 21us) op_driver.cc:348] REPLICATION: starting
1026 14:59:02.476968 (+ 848us) log.cc:844] Serialized 531 byte log entry
1026 14:59:02.830071 (+353103us) op_driver.cc:464] REPLICATION: finished
1026 14:59:03.252113 (+422042us) write_op.cc:301] APPLY: starting
1026 14:59:03.260245 (+ 8132us) tablet.cc:1365] starting BulkCheckPresence
1026 14:59:03.269827 (+ 9582us) tablet.cc:1368] finished BulkCheckPresence
1026 14:59:03.270367 (+ 540us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 14:59:03.272046 (+ 1679us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 14:59:03.272879 (+ 833us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1026 14:59:03.273263 (+ 384us) write_op.cc:312] APPLY: finished
1026 14:59:03.399143 (+125880us) log.cc:844] Serialized 22 byte log entry
1026 14:59:03.440839 (+ 41696us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:03.467368 (+ 26529us) write_op.cc:454] Released schema lock
1026 14:59:03.581468 (+114100us) write_op.cc:341] FINISH: Updating metrics
1026 14:59:03.622098 (+ 40630us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:03.622129 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":347644,"apply.run_cpu_time_us":4363,"apply.run_wall_time_us":382785,"num_ops":1,"prepare.queue_time_us":115251,"prepare.run_cpu_time_us":3028,"prepare.run_wall_time_us":147756,"replication_time_us":332532,"thread_start_us":283699,"threads_started":2}]]}
W20251026 14:59:03.900919 22728 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.008s sys 0.044s
I20251026 14:59:04.027184 22734 raft_consensus.cc:2468] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:59:04.023597 22650 rpcz_store.cc:276] 1026 14:59:02.771663 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:02.793761 (+ 22098us) service_pool.cc:225] Handling call
1026 14:59:02.817739 (+ 23978us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:02.818050 (+ 311us) raft_consensus.cc:1534] Early marking committed up to index 2
1026 14:59:03.036148 (+218098us) log.cc:844] Serialized 14 byte log entry
1026 14:59:03.989242 (+953094us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:03.989337 (+ 95us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:03.989403 (+ 66us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:03.989501 (+ 98us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:03.989702 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38442,"prepare.queue_time_us":122337,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":127061,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":33231,"spinlock_wait_cycles":197298432,"thread_start_us":1124678,"threads_started":3,"wal-append.queue_time_us":551786}
I20251026 14:59:04.034636 22732 raft_consensus.cc:2468] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
W20251026 14:59:04.032222 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:04.026358 22725 raft_consensus.cc:2410] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:04.035506 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:04.026744 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 207376us
I20251026 14:59:04.042641 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:04.023440 22727 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.711s user 0.024s sys 0.030s
I20251026 14:59:04.004007 23839 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:04.045496 22729 raft_consensus.cc:2468] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
W20251026 14:59:04.046964 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 104068us
W20251026 14:59:04.040462 22728 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.784s user 0.009s sys 0.044s
I20251026 14:59:04.048873 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:04.066300 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:04.068120 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:04.023550 22803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 814) took 1225 ms. Trace:
W20251026 14:59:04.074924 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:04.075594 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:04.075747 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:04.076164 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:04.076560 22725 raft_consensus.cc:2393] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:04.027272 23999 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:04.077581 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:04.078591 22725 raft_consensus.cc:2393] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:04.046885 23929 raft_consensus.cc:2804] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:04.068657 22734 raft_consensus.cc:2393] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:04.048496 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:04.084024 22732 raft_consensus.cc:2376] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3. Re-sending same reply.
W20251026 14:59:04.047796 23819 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.603s user 0.012s sys 0.010s
W20251026 14:59:04.073909 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:04.081058 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:04.081429 22731 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:04.068459 22804 raft_consensus.cc:2468] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:04.086951 22729 raft_consensus.cc:2376] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3. Re-sending same reply.
I20251026 14:59:04.098546 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:04.099692 22734 raft_consensus.cc:2393] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 4: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:04.063376 23980 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:04.105630 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:04.109740 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:04.109967 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:04.110363 22734 raft_consensus.cc:2376] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3. Re-sending same reply.
I20251026 14:59:04.108351 22622 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:04.113371 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 145914us
I20251026 14:59:04.028133 24010 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.078076 24007 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:04.116585 22729 raft_consensus.cc:1275] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20251026 14:59:04.086395 23819 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.645s user 0.013s sys 0.011s
I20251026 14:59:04.118811 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:04.119583 22725 raft_consensus.cc:2393] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:04.119161 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:04.114903 22773 leader_election.cc:336] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.572s (SENT)
W20251026 14:59:04.120579 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 75540.1us
I20251026 14:59:04.041760 23944 raft_consensus.cc:697] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:04.086274 23833 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:04.084797 23912 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:04.084597 24008 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:04.082363 23999 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:04.060974 23959 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.111642 23929 raft_consensus.cc:697] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:04.177486 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:04.177767 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:04.178432 22734 raft_consensus.cc:2393] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:04.033509 23641 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.121809 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:59:04.128136 22773 leader_election.cc:336] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.590s (SENT)
I20251026 14:59:04.128254 23912 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.174396 23944 consensus_queue.cc:237] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.175017 24010 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:04.175590 23980 consensus_queue.cc:1048] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 14:59:04.176061 22732 raft_consensus.cc:1275] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:59:04.074842 22803 rpcz_store.cc:276] 1026 14:59:02.797410 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:02.798700 (+ 1290us) service_pool.cc:225] Handling call
1026 14:59:04.022241 (+1223541us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:04.210868 22555 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50170 (request call id 57) took 1650 ms. Trace:
W20251026 14:59:04.173736 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 48322.3us
I20251026 14:59:04.208664 22729 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:04.127331 23833 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.218108 22773 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:04.222319 22731 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:04.099565 23839 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:04.243642 23916 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:04.123596 24004 raft_consensus.cc:2804] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:04.123339 23855 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:04.250653 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:04.270797 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:59:04.175043 23959 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:04.243757 23929 consensus_queue.cc:237] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.181180 23696 consensus_queue.cc:1048] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.142s
I20251026 14:59:04.124192 24007 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.324532 22800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 818) took 1517 ms. Trace:
I20251026 14:59:04.423943 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:04.271661 22732 raft_consensus.cc:1275] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20251026 14:59:04.322971 22656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 993) took 1559 ms. Trace:
W20251026 14:59:04.414942 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:04.425863 22656 rpcz_store.cc:276] 1026 14:59:02.763569 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:02.763750 (+ 181us) service_pool.cc:225] Handling call
1026 14:59:04.322757 (+1559007us) spinlock_profiling.cc:243] Waited 893 ms on lock 0x7b5400241654. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:04.322898 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2054544128}
I20251026 14:59:04.425838 22657 raft_consensus.cc:2410] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20251026 14:59:04.424170 23898 consensus_meta.cc:287] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.731s user 0.012s sys 0.002s
I20251026 14:59:04.323153 24010 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:04.427547 23898 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.431855 23839 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:04.432029 23898 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:04.428815 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:04.433132 22657 raft_consensus.cc:2410] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20251026 14:59:04.322719 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 69930.9us
I20251026 14:59:04.433434 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:04.272226 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:04.434192 22803 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:59:04.250547 23699 raft_consensus.cc:697] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:04.435379 22734 raft_consensus.cc:1275] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:59:04.429374 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:04.438659 23833 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:04.322053 23904 consensus_meta.cc:287] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.686s user 0.011s sys 0.001s
I20251026 14:59:04.429694 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:04.434497 23367 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:04.440040 22734 raft_consensus.cc:1275] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:59:04.440610 23980 raft_consensus.cc:2749] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251026 14:59:04.441372 22734 raft_consensus.cc:1275] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:59:04.424921 22800 rpcz_store.cc:276] 1026 14:59:02.807506 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:02.823440 (+ 15934us) service_pool.cc:225] Handling call
1026 14:59:02.823924 (+ 484us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:03.601594 (+777670us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:04.324161 (+722567us) spinlock_profiling.cc:243] Waited 613 ms on lock 0x7b54001bdd54. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:04.324425 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1409367296}
I20251026 14:59:04.442175 24007 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:04.442680 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 318568us
I20251026 14:59:04.347934 23641 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:04.429381 24025 raft_consensus.cc:2749] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:04.392666 24011 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:04.435561 23699 consensus_queue.cc:237] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.128728 22727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 996) took 1165 ms. Trace:
I20251026 14:59:04.434484 22725 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:04.251601 22555 rpcz_store.cc:276] 1026 14:59:02.559988 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:02.560149 (+ 161us) service_pool.cc:225] Handling call
1026 14:59:04.210751 (+1650602us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 14:59:02.597656 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 14:59:02.598417 (+ 761us) write_op.cc:432] Acquiring schema lock in shared mode
1026 14:59:02.598460 (+ 43us) write_op.cc:435] Acquired schema lock
1026 14:59:02.598488 (+ 28us) tablet.cc:660] Decoding operations
1026 14:59:02.598962 (+ 474us) write_op.cc:620] Acquiring the partition lock for write op
1026 14:59:02.599029 (+ 67us) write_op.cc:641] Partition lock acquired for write op
1026 14:59:02.599073 (+ 44us) tablet.cc:683] Acquiring locks for 1 operations
1026 14:59:02.599322 (+ 249us) tablet.cc:699] Row locks acquired
1026 14:59:02.599351 (+ 29us) write_op.cc:260] PREPARE: finished
1026 14:59:02.599532 (+ 181us) write_op.cc:270] Start()
1026 14:59:02.599703 (+ 171us) write_op.cc:276] Timestamp: P: 1761490742599466 usec, L: 0
1026 14:59:02.599735 (+ 32us) op_driver.cc:348] REPLICATION: starting
1026 14:59:02.600620 (+ 885us) log.cc:844] Serialized 526 byte log entry
1026 14:59:03.696033 (+1095413us) op_driver.cc:464] REPLICATION: finished
1026 14:59:03.757690 (+ 61657us) write_op.cc:301] APPLY: starting
1026 14:59:03.771230 (+ 13540us) tablet.cc:1365] starting BulkCheckPresence
1026 14:59:03.793658 (+ 22428us) tablet.cc:1368] finished BulkCheckPresence
1026 14:59:03.793802 (+ 144us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 14:59:03.821082 (+ 27280us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 14:59:03.821245 (+ 163us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1026 14:59:03.821342 (+ 97us) write_op.cc:312] APPLY: finished
1026 14:59:03.821557 (+ 215us) log.cc:844] Serialized 22 byte log entry
1026 14:59:03.821742 (+ 185us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:03.821822 (+ 80us) write_op.cc:454] Released schema lock
1026 14:59:03.821876 (+ 54us) write_op.cc:341] FINISH: Updating metrics
1026 14:59:03.822240 (+ 364us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:03.822271 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":18860,"apply.run_cpu_time_us":2110,"apply.run_wall_time_us":79627,"num_ops":1,"prepare.queue_time_us":152,"prepare.run_cpu_time_us":3510,"prepare.run_wall_time_us":17831,"replication_time_us":1096108}]]}
I20251026 14:59:04.436913 22729 raft_consensus.cc:1275] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:59:04.451761 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:04.437875 22732 raft_consensus.cc:1275] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:59:04.438109 22808 raft_consensus.cc:2468] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:04.449440 22774 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:04.434217 24010 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.455139 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:04.455871 22657 raft_consensus.cc:2393] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:04.456298 24010 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:04.428174 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 300050us
I20251026 14:59:04.342561 23912 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:04.449520 23959 raft_consensus.cc:493] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:04.457537 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:04.458076 22657 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 3
I20251026 14:59:04.457729 23959 raft_consensus.cc:515] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:04.459939 23959 leader_election.cc:290] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:04.250938 22728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 977) took 1337 ms. Trace:
I20251026 14:59:04.341405 24024 consensus_queue.cc:1059] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.111s
I20251026 14:59:04.454437 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:04.461059 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:04.463846 24010 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:04.464465 24010 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.465628 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:04.466504 22659 raft_consensus.cc:2393] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 4: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:04.465260 23959 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:04.467866 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:04.468603 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:04.468379 23959 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.470276 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:04.470369 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:04.471156 22729 raft_consensus.cc:2368] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3.
I20251026 14:59:04.471226 22654 raft_consensus.cc:2410] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:04.469709 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:04.468138 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:04.472173 22732 raft_consensus.cc:2368] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3.
I20251026 14:59:04.472376 22656 raft_consensus.cc:3060] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:04.473170 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:04.465159 22657 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
W20251026 14:59:04.414587 23985 consensus_meta.cc:287] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1: Time spent flushing consensus metadata: real 0.719s user 0.004s sys 0.010s
W20251026 14:59:04.277376 24019 log.cc:927] Time spent T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.164s user 0.000s sys 0.001s
I20251026 14:59:04.480373 23959 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:04.481405 22725 raft_consensus.cc:2393] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 2: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:04.481616 22731 raft_consensus.cc:1240] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
I20251026 14:59:04.486662 22731 raft_consensus.cc:1240] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251026 14:59:04.481915 22697 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:04.487773 22731 raft_consensus.cc:1240] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
I20251026 14:59:04.487581 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:04.489020 22653 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
W20251026 14:59:04.490756 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:04.491489 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:04.492120 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:04.493134 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:04.246093 23941 consensus_queue.cc:1059] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.236s
I20251026 14:59:04.180497 24008 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.503257 24008 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:04.472543 24010 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:04.503607 22734 raft_consensus.cc:1240] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:04.496522 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:04.504241 22732 raft_consensus.cc:2368] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3.
I20251026 14:59:04.479132 24011 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.505322 22659 raft_consensus.cc:2393] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:04.507114 22734 raft_consensus.cc:1240] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251026 14:59:04.508477 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:04.509584 22732 raft_consensus.cc:2410] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:04.449118 23855 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.488194 24028 raft_consensus.cc:2804] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:04.512346 23855 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:04.513965 22734 raft_consensus.cc:1240] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251026 14:59:04.450150 22727 rpcz_store.cc:276] 1026 14:59:02.956566 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:02.967142 (+ 10576us) service_pool.cc:225] Handling call
1026 14:59:02.967472 (+ 330us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:02.967923 (+ 451us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:02.967954 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:02.968477 (+ 523us) log.cc:844] Serialized 26 byte log entry
1026 14:59:03.239215 (+270738us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:04.065791 (+826576us) log.cc:844] Serialized 14 byte log entry
1026 14:59:04.083504 (+ 17713us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:04.098815 (+ 15311us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:04.110541 (+ 11726us) raft_consensus.cc:1685] finished
1026 14:59:04.110560 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:04.122250 (+ 11690us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2276,"prepare.queue_time_us":383290,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":27,"raft.queue_time_us":257,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":5668,"thread_start_us":1009753,"threads_started":2,"wal-append.queue_time_us":362073}
I20251026 14:59:04.516669 22727 raft_consensus.cc:1240] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251026 14:59:04.494819 23819 raft_consensus.cc:2804] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:04.518383 23819 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:04.518850 23819 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:04.496095 22657 raft_consensus.cc:2393] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
W20251026 14:59:04.498760 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:04.499096 24004 raft_consensus.cc:697] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:04.522532 24004 consensus_queue.cc:237] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.524690 22730 raft_consensus.cc:1240] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20251026 14:59:04.499928 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:04.500257 22725 raft_consensus.cc:1240] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
I20251026 14:59:04.525851 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:04.526445 22730 raft_consensus.cc:2468] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:59:04.526768 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:04.527225 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:04.523080 23911 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:04.528334 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:04.528970 22730 raft_consensus.cc:2368] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3.
I20251026 14:59:04.499533 23740 raft_consensus.cc:3060] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:04.527324 22725 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:59:04.531941 22656 raft_consensus.cc:2410] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20251026 14:59:04.532714 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:04.539535 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:04.540220 22732 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:04.541641 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:04.541975 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:04.542625 22654 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:04.544831 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:04.546139 22808 raft_consensus.cc:2393] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:04.549547 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:04.550590 22804 raft_consensus.cc:2393] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:04.552865 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:04.448745 23916 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.566712 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:04.569181 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:04.571486 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:04.572079 22730 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:04.572259 22656 raft_consensus.cc:1240] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:04.557029 22700 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:04.547706 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:04.559355 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:04.495784 23944 raft_consensus.cc:3055] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:04.574148 22800 raft_consensus.cc:2410] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:04.574301 22657 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 5
I20251026 14:59:04.575264 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:04.576020 22800 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:04.577121 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:04.578601 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:04.579505 22800 raft_consensus.cc:2368] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 3. Current term is 4.
I20251026 14:59:04.579782 22657 raft_consensus.cc:1275] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251026 14:59:04.580655 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:04.569872 22729 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:59:04.582134 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:04.581305 22803 raft_consensus.cc:1275] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20251026 14:59:04.582789 22803 pending_rounds.cc:85] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821: Aborting all ops after (but not including) 1
I20251026 14:59:04.580396 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:04.581374 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:04.586232 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:04.586696 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:04.543130 23855 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:04.591029 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:04.591259 22656 raft_consensus.cc:1275] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:59:04.537810 23839 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:04.593726 22651 raft_consensus.cc:1275] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:59:04.594311 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:04.587868 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:04.595068 22729 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:04.596014 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:04.588804 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:04.588845 22803 pending_rounds.cc:107] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20251026 14:59:04.597702 22803 raft_consensus.cc:2889] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:04.598606 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:04.599666 22657 raft_consensus.cc:2393] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 2: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:04.600611 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:04.584625 22800 raft_consensus.cc:2376] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3. Re-sending same reply.
I20251026 14:59:04.538455 24028 raft_consensus.cc:697] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
W20251026 14:59:04.590018 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:04.554804 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:04.607903 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:04.554194 23959 consensus_queue.cc:1059] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20251026 14:59:04.617779 22804 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:04.609530 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:04.611577 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:04.614269 22657 raft_consensus.cc:1275] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:59:04.615396 22653 raft_consensus.cc:2410] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:04.616382 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:04.581676 23944 raft_consensus.cc:740] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
W20251026 14:59:04.619195 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 62292us
I20251026 14:59:04.634605 22700 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:04.633651 22656 raft_consensus.cc:2393] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:04.585717 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:04.585686 22802 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:04.644330 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:04.660616 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:04.668743 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 192398us
W20251026 14:59:04.669127 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 212037us
W20251026 14:59:04.669874 22624 leader_election.cc:336] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.633s (SENT)
I20251026 14:59:04.616122 24010 raft_consensus.cc:2749] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:04.672879 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:04.676183 22732 raft_consensus.cc:2468] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:04.676532 23999 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.677949 24011 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:04.678877 22655 raft_consensus.cc:1275] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:59:04.680045 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:04.680699 22655 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
W20251026 14:59:04.642282 24018 log.cc:927] Time spent T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20251026 14:59:04.677211 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:04.690192 22773 leader_election.cc:336] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.683s (SENT)
I20251026 14:59:04.690728 22624 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:04.691687 22773 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:04.695160 22774 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:04.695737 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:04.696250 22654 raft_consensus.cc:2410] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:04.696097 22658 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:04.605170 23855 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.605713 23839 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.747882 22653 raft_consensus.cc:2393] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:04.748353 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:04.753333 22732 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:04.758054 22729 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:04.751211 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:04.752067 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 82231.1us
I20251026 14:59:04.763602 22658 raft_consensus.cc:1275] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20251026 14:59:04.763947 22624 leader_election.cc:336] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.647s (SENT)
I20251026 14:59:04.764333 22624 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:04.758993 22804 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
W20251026 14:59:04.749564 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 79338.6us
I20251026 14:59:04.767319 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:04.767963 22729 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:04.669801 22727 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:04.702049 24031 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:04.747073 24012 raft_consensus.cc:2749] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:04.777376 22697 leader_election.cc:400] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 3. Current term is 4.
I20251026 14:59:04.777874 22697 leader_election.cc:403] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20251026 14:59:04.752476 23929 raft_consensus.cc:2804] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:59:04.753734 24008 raft_consensus.cc:2749] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:59:04.797416 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:04.762394 22732 raft_consensus.cc:2468] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:04.783154 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:04.801405 22650 raft_consensus.cc:1240] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251026 14:59:04.790961 22624 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:04.793723 22733 raft_consensus.cc:2410] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20251026 14:59:04.864656 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 230135us
W20251026 14:59:04.780740 22773 leader_election.cc:336] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.850s (SENT)
I20251026 14:59:04.868466 22773 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:04.460474 22728 rpcz_store.cc:276] 1026 14:59:02.913741 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:02.919053 (+ 5312us) service_pool.cc:225] Handling call
1026 14:59:02.938306 (+ 19253us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:02.938808 (+ 502us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:02.938846 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:02.939428 (+ 582us) log.cc:844] Serialized 26 byte log entry
1026 14:59:03.157469 (+218041us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:03.373146 (+215677us) spinlock_profiling.cc:243] Waited 3.4 ms on lock 0x7b1800137440. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce0b1992 00007fd8ce0b3d61 00007fd8ce0b3d19 00007fd8ce0b3ca9 00007fd8ce0b3c71 00007fd8ce0b2f6d 0000000000379254 0000000000379099 00007fd8ce094d1a
1026 14:59:04.084929 (+711783us) log.cc:844] Serialized 14 byte log entry
1026 14:59:04.110711 (+ 25782us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:04.117204 (+ 6493us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:04.121693 (+ 4489us) raft_consensus.cc:1685] finished
1026 14:59:04.208248 (+ 86555us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:04.247915 (+ 39667us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12354,"prepare.queue_time_us":441176,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":8813,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":7916,"spinlock_wait_cycles":7810304,"thread_start_us":954152,"threads_started":2,"wal-append.queue_time_us":310021}
I20251026 14:59:04.865890 22700 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:04.898249 22654 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:04.898617 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:04.784032 22697 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:04.900199 22729 raft_consensus.cc:2393] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:04.784747 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:04.785216 23740 raft_consensus.cc:2749] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:04.782876 24007 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:04.782658 24032 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:04.789806 23912 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:04.904412 22734 raft_consensus.cc:1240] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251026 14:59:04.771536 24011 raft_consensus.cc:2749] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:59:04.796167 23402 raft_consensus.cc:1240] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:04.802078 22655 raft_consensus.cc:1275] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20251026 14:59:04.908689 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 117839us
W20251026 14:59:04.784646 24001 log.cc:927] Time spent T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 1.156s user 0.005s sys 0.012s
I20251026 14:59:04.820946 22730 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:04.909237 22734 raft_consensus.cc:1240] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251026 14:59:04.780103 24031 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.911724 23912 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.837198 23916 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:04.873814 23765 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:04.802661 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:04.912628 22650 raft_consensus.cc:1275] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20251026 14:59:04.899701 24012 raft_consensus.cc:2749] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251026 14:59:04.902400 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 121752us
I20251026 14:59:04.802404 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:04.914765 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:04.785734 23999 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:04.795004 23839 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:04.928689 22729 raft_consensus.cc:1240] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251026 14:59:04.798296 23855 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:04.920898 22730 raft_consensus.cc:1240] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251026 14:59:04.760962 23944 consensus_queue.cc:260] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.923893 22654 raft_consensus.cc:2410] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:04.899232 23929 raft_consensus.cc:697] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:04.932874 22730 raft_consensus.cc:1240] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251026 14:59:04.915215 23740 raft_consensus.cc:2749] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 3. Current term is 4.
I20251026 14:59:04.948228 22624 leader_election.cc:400] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507) with higher term. Message: Invalid argument: T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3.
I20251026 14:59:04.948673 22624 leader_election.cc:403] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20251026 14:59:04.942031 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:04.949887 22650 raft_consensus.cc:2393] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:04.946480 22622 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:04.947147 22730 raft_consensus.cc:1240] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251026 14:59:04.903983 23845 raft_consensus.cc:2749] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:59:04.903164 24007 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:04.779716 24028 consensus_queue.cc:237] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.923813 23765 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:04.960465 23740 raft_consensus.cc:2749] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:04.975467 22622 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:04.977643 22654 raft_consensus.cc:1240] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:04.903630 24032 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:04.958250 23916 raft_consensus.cc:2804] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:04.982144 22622 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:04.963507 23997 log.cc:927] Time spent T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 1.340s user 0.009s sys 0.024s
I20251026 14:59:04.968324 22773 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:04.956240 23944 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:04.992480 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:04.972455 24004 raft_consensus.cc:2749] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507) with higher term. Message: Invalid argument: T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3.
W20251026 14:59:04.993366 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 127631us
I20251026 14:59:04.972847 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:04.959472 23941 consensus_queue.cc:1048] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20251026 14:59:04.997123 22700 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:04.985644 23696 raft_consensus.cc:2749] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20251026 14:59:05.002568 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:04.983316 23959 raft_consensus.cc:2804] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:04.984937 23916 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:05.004091 22622 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:04.985932 23855 raft_consensus.cc:2804] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:04.988713 22728 raft_consensus.cc:1240] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251026 14:59:05.009654 22654 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:05.010301 22659 raft_consensus.cc:2410] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20251026 14:59:05.000192 24039 log.cc:927] Time spent T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20251026 14:59:05.012349 22624 leader_election.cc:400] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 2 election: Vote denied by peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507) with higher term. Message: Invalid argument: T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3.
W20251026 14:59:05.015965 23993 log.cc:927] Time spent T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 1.365s user 0.002s sys 0.013s
I20251026 14:59:05.021380 24042 consensus_queue.cc:1048] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20251026 14:59:05.022707 22622 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:05.023945 23911 raft_consensus.cc:2804] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:05.024449 23911 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:05.024899 23911 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:05.029968 24004 raft_consensus.cc:2804] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:05.030606 24004 raft_consensus.cc:697] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:05.030742 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 982) took 2078 ms. Trace:
I20251026 14:59:05.031509 24004 consensus_queue.cc:237] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.031872 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:05.031105 22726 rpcz_store.cc:276] 1026 14:59:02.952002 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:02.952139 (+ 137us) service_pool.cc:225] Handling call
1026 14:59:02.952557 (+ 418us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:02.953016 (+ 459us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:02.953055 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:02.953599 (+ 544us) log.cc:844] Serialized 26 byte log entry
1026 14:59:03.072171 (+118572us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:03.239360 (+167189us) log.cc:844] Serialized 14 byte log entry
1026 14:59:03.401647 (+162287us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:03.401723 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:04.690778 (+1289055us) spinlock_profiling.cc:243] Waited 78.6 ms on lock 0x7b440002304c. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8cdfc0905 00007fd8cdfc1322 00007fd8d0d0a231 00007fd8d0d05b5a 00007fd8d0cf991e 00007fd8d0cf8c66 00007fd8d0cf8888 00007fd8d0d022c1 00007fd8d0d02279 00007fd8d0d02209
1026 14:59:05.030414 (+339636us) raft_consensus.cc:1685] finished
1026 14:59:05.030519 (+ 105us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:05.030676 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":352347,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":368804,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":7003,"spinlock_wait_cycles":180865920,"thread_start_us":344213,"threads_started":3,"wal-append.queue_time_us":631243}
I20251026 14:59:04.980849 23641 consensus_queue.cc:1059] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
W20251026 14:59:05.039556 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:05.001385 23904 raft_consensus.cc:2804] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251026 14:59:05.042169 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 258163us
I20251026 14:59:05.042193 22658 raft_consensus.cc:1240] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:05.044818 22773 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.047679 24042 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:05.009836 23916 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
W20251026 14:59:05.037559 24023 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.021s sys 0.030s
W20251026 14:59:05.052021 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 44518.1us
I20251026 14:59:05.033712 22727 raft_consensus.cc:1240] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: []
W20251026 14:59:05.053169 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 56155.2us
I20251026 14:59:05.054073 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:05.037688 23855 raft_consensus.cc:697] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:05.062731 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:05.063890 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:05.060168 23740 raft_consensus.cc:2749] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:59:05.064643 22658 raft_consensus.cc:2393] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
W20251026 14:59:05.057618 24023 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.524s user 0.022s sys 0.030s
I20251026 14:59:05.065461 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:05.065762 22697 leader_election.cc:304] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:05.065963 22773 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.066478 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:05.067014 22773 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.054224 23959 raft_consensus.cc:697] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:05.068374 23696 raft_consensus.cc:2764] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
W20251026 14:59:05.068923 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:05.068645 23959 consensus_queue.cc:237] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:59:05.003787 23929 consensus_queue.cc:237] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.040709 23819 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.043085 23904 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:05.032948 22655 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:05.059985 22624 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:05.036794 23912 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:05.069156 22773 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:05.049551 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:05.100858 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:05.106034 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:05.106802 22732 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:05.107241 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:05.108062 22731 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
W20251026 14:59:05.108062 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 145754us
I20251026 14:59:05.108994 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:05.109807 22732 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:05.110064 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:05.067039 22658 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:05.110616 22731 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:05.111611 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:05.111976 22658 raft_consensus.cc:1275] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20251026 14:59:05.112138 22731 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:59:05.112044 22700 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.113158 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:05.069000 23696 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:05.117996 22731 raft_consensus.cc:2468] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:59:05.126538 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:05.129892 22725 raft_consensus.cc:2393] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:05.130532 22773 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.062160 23975 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:05.077913 24007 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:05.074420 24044 raft_consensus.cc:2749] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:05.134476 22700 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.102577 23904 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:05.134452 22773 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.143652 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:05.144687 22731 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:05.144795 22624 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:05.144795 22732 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:05.114761 23951 raft_consensus.cc:2749] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:05.147641 22624 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:05.148563 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:05.149128 22732 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:59:05.149824 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:05.071962 23740 consensus_queue.cc:1048] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20251026 14:59:05.124611 23898 raft_consensus.cc:2749] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:05.124815 23912 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:05.172114 22700 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.167207 22732 raft_consensus.cc:2468] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:05.129199 24008 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:05.163985 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:05.133826 23934 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:05.175415 22731 raft_consensus.cc:2376] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3. Re-sending same reply.
I20251026 14:59:05.175935 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:05.176318 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:05.146395 23833 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:05.145840 24012 raft_consensus.cc:2804] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:05.180819 22732 raft_consensus.cc:1275] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20251026 14:59:05.145342 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 56180.9us
I20251026 14:59:05.182574 22700 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.166841 24007 raft_consensus.cc:2749] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:05.129406 23941 consensus_queue.cc:1059] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20251026 14:59:05.202787 22700 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.176183 24028 raft_consensus.cc:2804] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:59:05.170312 24051 raft_consensus.cc:2749] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20251026 14:59:05.209350 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 74967.5us
I20251026 14:59:05.146250 23975 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:05.184623 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:05.212908 22773 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.063199 23855 consensus_queue.cc:237] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.179942 24050 raft_consensus.cc:2749] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:05.199230 24044 raft_consensus.cc:2749] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:05.220551 22773 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.209543 24012 raft_consensus.cc:697] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:05.185293 23912 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:05.210592 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 101079us
I20251026 14:59:05.208786 23999 raft_consensus.cc:2749] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251026 14:59:05.244187 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 96607.6us
I20251026 14:59:05.185493 23934 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.250918 23934 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:05.246363 22773 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.213836 24045 raft_consensus.cc:2749] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251026 14:59:05.188633 24008 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.197821 23833 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.247017 23912 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:05.217227 23641 raft_consensus.cc:3055] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:05.218912 24051 raft_consensus.cc:2749] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:05.218578 24028 raft_consensus.cc:697] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:05.221143 23941 raft_consensus.cc:2804] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:05.224813 22727 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:05.259531 23951 raft_consensus.cc:2749] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:05.303244 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:59:05.304404 24058 log.cc:927] Time spent T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20251026 14:59:05.306386 22806 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:05.278402 23641 raft_consensus.cc:740] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:05.278083 23916 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.260947 23839 consensus_queue.cc:1059] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
W20251026 14:59:05.312595 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 66302.1us
I20251026 14:59:05.298552 23819 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:05.293411 24062 raft_consensus.cc:2804] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:05.324445 22773 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.328720 22773 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.296684 23934 consensus_queue.cc:1059] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20251026 14:59:05.296712 23912 consensus_queue.cc:1059] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20251026 14:59:05.305030 23898 consensus_queue.cc:1059] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20251026 14:59:05.312827 24011 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
W20251026 14:59:05.335507 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 132825us
I20251026 14:59:05.339062 23898 raft_consensus.cc:3055] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:05.339506 23898 raft_consensus.cc:740] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:05.340368 23898 consensus_queue.cc:260] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.341468 23839 raft_consensus.cc:3055] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:05.341869 23839 raft_consensus.cc:740] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:05.342023 24008 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:05.342651 23839 consensus_queue.cc:260] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.343986 23839 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
W20251026 14:59:05.344238 24034 log.cc:927] Time spent T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.833s user 0.006s sys 0.000s
I20251026 14:59:05.320523 24062 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:05.338157 22700 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.320801 23951 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:05.349484 22700 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.330446 24064 raft_consensus.cc:3055] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:05.351751 23641 consensus_queue.cc:260] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.352733 22806 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:05.345571 24062 raft_consensus.cc:3060] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:05.356500 23929 raft_consensus.cc:3060] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:05.356724 22700 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.356966 23740 raft_consensus.cc:2749] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:05.362008 22697 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:05.354338 23898 raft_consensus.cc:3060] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:05.364701 24063 consensus_queue.cc:1059] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
I20251026 14:59:05.367338 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:05.346719 24012 consensus_queue.cc:237] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.364250 23833 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:05.350486 23819 raft_consensus.cc:2749] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20251026 14:59:05.377465 24057 log.cc:927] Time spent T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.093s user 0.001s sys 0.000s
I20251026 14:59:05.379850 22700 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.381012 22700 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.358985 24064 raft_consensus.cc:740] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:05.388789 22700 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.346513 24028 consensus_queue.cc:237] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.374145 23641 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:05.400987 22700 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.381165 23833 raft_consensus.cc:3055] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:05.348379 24011 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:05.362171 23941 raft_consensus.cc:697] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:05.379664 24044 raft_consensus.cc:2749] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:05.360817 23951 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.385824 24050 raft_consensus.cc:2804] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:05.411633 22700 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.399153 23740 raft_consensus.cc:2749] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:59:05.398367 23975 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:05.416704 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 88064.2us
I20251026 14:59:05.402648 24045 raft_consensus.cc:2749] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:05.404764 23819 raft_consensus.cc:2749] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:05.409005 23833 raft_consensus.cc:740] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:05.440263 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:05.453889 22804 raft_consensus.cc:2468] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 3.
I20251026 14:59:05.385509 23904 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.446015 23941 consensus_queue.cc:237] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.446569 23951 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:05.446623 23696 raft_consensus.cc:2749] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:05.483060 24064 consensus_queue.cc:260] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.483820 23916 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:05.484499 24064 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:05.447317 22802 raft_consensus.cc:2410] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:05.449390 24011 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:05.489573 22804 raft_consensus.cc:1240] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251026 14:59:05.451031 24024 raft_consensus.cc:3055] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:05.451440 24044 raft_consensus.cc:2749] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:05.385453 23911 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.451516 22773 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:05.438563 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:05.454471 24050 raft_consensus.cc:697] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:05.477440 22656 raft_consensus.cc:1240] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: []
W20251026 14:59:05.479928 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:05.455057 23944 consensus_queue.cc:1048] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20251026 14:59:05.447492 24066 raft_consensus.cc:2749] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:05.449785 22658 raft_consensus.cc:2468] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 2.
I20251026 14:59:05.491595 22728 raft_consensus.cc:1275] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20251026 14:59:05.492779 22731 raft_consensus.cc:1275] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20251026 14:59:05.493250 22733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 998) took 2528 ms (client timeout 1780 ms). Trace:
I20251026 14:59:05.498365 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:05.506342 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:05.506206 22733 rpcz_store.cc:269] 1026 14:59:02.957188 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:02.958910 (+ 1722us) service_pool.cc:225] Handling call
1026 14:59:03.302269 (+343359us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b5800250598. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8ce074451 00007fd8ce074302 00007fd8d6f0dad9 00007fd8d6090d30 00007fd8d615984b 00007fd8d61598d1 00007fd8d6107660 00007fd8d6106f44 00007fd8cf6a4834 00007fd8cf6a47c2
1026 14:59:05.460876 (+2158607us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b54000d66d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:05.485619 (+ 24743us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3449516800}
I20251026 14:59:05.502628 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:05.503058 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:05.509184 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:05.509446 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:05.510319 22807 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:05.511457 23911 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:05.509474 23916 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:05.514248 23916 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.514927 24031 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:05.492168 24024 raft_consensus.cc:740] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:05.516286 23916 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:05.505250 22728 pending_rounds.cc:85] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e: Aborting all ops after (but not including) 1
I20251026 14:59:05.517165 22728 pending_rounds.cc:107] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20251026 14:59:05.517499 22728 raft_consensus.cc:2889] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:05.516768 24024 consensus_queue.cc:260] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:05.518362 22652 raft_consensus.cc:1275] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:59:05.518759 24024 raft_consensus.cc:3060] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:05.509071 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:05.519760 22799 raft_consensus.cc:2468] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:05.520028 24032 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:05.521206 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:05.523836 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 72401.5us
I20251026 14:59:05.526980 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:05.527586 22803 raft_consensus.cc:2393] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 4: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:05.491427 23833 consensus_queue.cc:260] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.567405 23833 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:05.534461 23696 raft_consensus.cc:2764] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20251026 14:59:05.538056 24030 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:05.568930 24030 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.570995 24030 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:05.514948 24010 raft_consensus.cc:3055] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:05.571621 24010 raft_consensus.cc:740] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:05.571553 24062 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.572765 22806 raft_consensus.cc:1240] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:05.538906 23934 raft_consensus.cc:3055] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:05.539359 23765 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:05.539841 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 814) took 1089 ms. Trace:
I20251026 14:59:05.540416 23951 consensus_queue.cc:1048] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:59:05.577311 22651 rpcz_store.cc:276] 1026 14:59:04.450354 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:04.450847 (+ 493us) service_pool.cc:225] Handling call
1026 14:59:04.451275 (+ 428us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:04.594233 (+142958us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:05.539552 (+945319us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b540034c954. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:05.539773 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":326589184}
W20251026 14:59:05.542883 24077 log.cc:927] Time spent T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.097s user 0.002s sys 0.000s
W20251026 14:59:05.548379 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:05.579070 22622 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:05.550159 24050 consensus_queue.cc:237] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.504875 24056 consensus_queue.cc:1059] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
W20251026 14:59:05.552325 24074 log.cc:927] Time spent T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20251026 14:59:05.554190 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:05.594112 22657 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:05.557664 22729 raft_consensus.cc:1240] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251026 14:59:05.564510 23904 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:05.536159 24011 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:05.566514 23929 raft_consensus.cc:2749] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251026 14:59:05.575670 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 49828.1us
I20251026 14:59:05.605477 22773 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:05.582773 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 87528.7us
W20251026 14:59:05.584276 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 172730us
I20251026 14:59:05.588989 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:05.608618 22700 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.609766 22700 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.610635 22700 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.597443 22729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 582) took 1079 ms. Trace:
I20251026 14:59:05.566421 22728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 594) took 1045 ms. Trace:
I20251026 14:59:05.497753 23367 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters:
I20251026 14:59:05.576992 23934 raft_consensus.cc:740] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:05.611151 22729 rpcz_store.cc:276] 1026 14:59:04.517529 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:04.929787 (+412258us) service_pool.cc:225] Handling call
1026 14:59:05.557118 (+627331us) spinlock_profiling.cc:243] Waited 3.31 ms on lock 0x7b5800250598. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8ce074451 00007fd8ce074302 00007fd8d6f0dad9 00007fd8d6090d30 00007fd8d615984b 00007fd8d61598d1 00007fd8d611b580 00007fd8d6105ee9 00007fd8d6105992 00007fd8cf69fd34
1026 14:59:05.557388 (+ 270us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:05.595391 (+ 38003us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:05.597383 (+ 1992us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7607168}
I20251026 14:59:05.587618 24030 raft_consensus.cc:2749] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:05.614856 22700 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.586269 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:05.616918 22774 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.607235 23951 consensus_queue.cc:1048] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.011s
W20251026 14:59:05.623644 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:05.620333 22700 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.587265 23696 consensus_queue.cc:1059] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20251026 14:59:05.597510 24070 raft_consensus.cc:2749] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:05.633124 22700 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.634855 22700 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:05.639003 22700 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:05.612497 23765 raft_consensus.cc:2749] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:05.622460 24056 raft_consensus.cc:2749] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:05.662246 22806 raft_consensus.cc:1240] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:05.609004 24011 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.617229 24081 raft_consensus.cc:2749] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:05.679504 22554 catalog_manager.cc:5649] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 1 to 3, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 3 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:05.618876 24066 raft_consensus.cc:2749] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:05.607110 24073 consensus_queue.cc:1048] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
I20251026 14:59:05.684366 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:05.639796 23929 raft_consensus.cc:2749] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:05.663581 22552 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38878 (request call id 162) took 3308 ms. Trace:
I20251026 14:59:05.686935 22554 catalog_manager.cc:5649] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 14:59:05.626332 23400 raft_consensus.cc:2468] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
W20251026 14:59:05.630712 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 141995us
I20251026 14:59:05.641906 24051 raft_consensus.cc:2804] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:05.669493 23765 raft_consensus.cc:2804] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:05.669888 24026 consensus_queue.cc:1048] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
W20251026 14:59:05.671291 24078 log.cc:927] Time spent T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20251026 14:59:05.671896 24044 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:05.676234 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:05.710050 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:05.711187 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:05.677454 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:05.666028 23696 raft_consensus.cc:3055] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:05.678776 24011 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:05.716569 22622 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:05.679416 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:05.679651 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:05.630113 23819 raft_consensus.cc:2749] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251026 14:59:05.680682 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 62308.3us
I20251026 14:59:05.682051 22651 raft_consensus.cc:1275] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:59:05.639822 23740 raft_consensus.cc:2749] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:59:05.689417 22808 raft_consensus.cc:2393] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
W20251026 14:59:05.684731 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 67927.8us
I20251026 14:59:05.689736 22552 rpcz_store.cc:276] 1026 14:59:02.352353 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:02.352575 (+ 222us) service_pool.cc:225] Handling call
1026 14:59:05.660655 (+3308080us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 14:59:02.952170 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 14:59:02.952829 (+ 659us) write_op.cc:432] Acquiring schema lock in shared mode
1026 14:59:02.952870 (+ 41us) write_op.cc:435] Acquired schema lock
1026 14:59:02.952892 (+ 22us) tablet.cc:660] Decoding operations
1026 14:59:02.954676 (+ 1784us) write_op.cc:620] Acquiring the partition lock for write op
1026 14:59:02.954721 (+ 45us) write_op.cc:641] Partition lock acquired for write op
1026 14:59:02.954750 (+ 29us) tablet.cc:683] Acquiring locks for 18 operations
1026 14:59:02.956101 (+ 1351us) tablet.cc:699] Row locks acquired
1026 14:59:02.956120 (+ 19us) write_op.cc:260] PREPARE: finished
1026 14:59:02.956273 (+ 153us) write_op.cc:270] Start()
1026 14:59:02.956399 (+ 126us) write_op.cc:276] Timestamp: P: 1761490742956227 usec, L: 0
1026 14:59:02.956428 (+ 29us) op_driver.cc:348] REPLICATION: starting
1026 14:59:03.686013 (+729585us) spinlock_profiling.cc:243] Waited 2.67 ms on lock 0x7b54003af820. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d5e27964 00007fd8d10d54fb 00007fd8d10d524a 00007fd8d1159505 00007fd8d11599a9 00007fd8d4b8bce1 00007fd8d4b8b4b8 00007fd8d4b90151 00007fd8d4b90109 00007fd8d4b90099 00007fd8d4b90061
1026 14:59:03.686739 (+ 726us) log.cc:844] Serialized 6962 byte log entry
1026 14:59:03.687797 (+ 1058us) spinlock_profiling.cc:243] Waited 84.7 ms on lock 0x7b54003af554. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d11599c8 00007fd8d4b8bce1 00007fd8d4b8b4b8 00007fd8d4b90151 00007fd8d4b90109 00007fd8d4b90099 00007fd8d4b90061 00007fd8d4b8f28d 0000000000379254 0000000000379099
1026 14:59:03.699234 (+ 11437us) op_driver.cc:464] REPLICATION: finished
1026 14:59:03.955363 (+256129us) write_op.cc:301] APPLY: starting
1026 14:59:03.966848 (+ 11485us) tablet.cc:1365] starting BulkCheckPresence
1026 14:59:03.967360 (+ 512us) tablet.cc:1368] finished BulkCheckPresence
1026 14:59:03.967389 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 14:59:04.098356 (+130967us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 14:59:04.206151 (+107795us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=18
1026 14:59:04.206230 (+ 79us) write_op.cc:312] APPLY: finished
1026 14:59:04.206821 (+ 591us) log.cc:844] Serialized 124 byte log entry
1026 14:59:04.246393 (+ 39572us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:04.319119 (+ 72726us) write_op.cc:454] Released schema lock
1026 14:59:04.329898 (+ 10779us) write_op.cc:341] FINISH: Updating metrics
1026 14:59:04.385457 (+ 55559us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:04.386736 (+ 1279us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1324,"reported_tablets":30,"child_traces":[["op",{"apply.queue_time_us":248523,"apply.run_cpu_time_us":6333,"apply.run_wall_time_us":419268,"num_ops":18,"prepare.queue_time_us":367,"prepare.run_cpu_time_us":26880,"prepare.run_wall_time_us":735779,"replication_time_us":742617,"spinlock_wait_cycles":200931584,"thread_start_us":342569,"threads_started":1}]]}
W20251026 14:59:05.701982 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:05.625553 23699 consensus_queue.cc:1048] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.058s
I20251026 14:59:05.704916 22554 catalog_manager.cc:5649] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 1 to 3, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 3 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:05.618800 24023 consensus_queue.cc:1048] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20251026 14:59:05.727212 22554 catalog_manager.cc:5649] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 1 to 3, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 3 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
W20251026 14:59:05.718868 24042 consensus_meta.cc:287] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.669s user 0.016s sys 0.000s
I20251026 14:59:05.716044 23696 raft_consensus.cc:740] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:05.613408 22728 rpcz_store.cc:276] 1026 14:59:04.520549 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:05.014896 (+494347us) service_pool.cc:225] Handling call
1026 14:59:05.485326 (+470430us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:05.518029 (+ 32703us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:05.566170 (+ 48141us) spinlock_profiling.cc:243] Waited 466 ms on lock 0x7b54000d66d0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:05.566347 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1072183040}
I20251026 14:59:05.706161 23765 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:05.724053 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:05.706660 24051 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:05.720669 24011 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:05.761538 24011 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.763749 24011 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:05.717618 22800 raft_consensus.cc:2393] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:05.764336 24051 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:05.636605 24012 consensus_queue.cc:1059] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
I20251026 14:59:05.729120 22808 raft_consensus.cc:1275] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:05.730393 24075 consensus_queue.cc:1048] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20251026 14:59:05.642042 24045 consensus_queue.cc:1048] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.077s
I20251026 14:59:05.715571 24062 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:05.720818 23904 raft_consensus.cc:2804] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:05.734920 22622 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:05.731693 24084 consensus_queue.cc:1048] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:05.769841 23403 raft_consensus.cc:2393] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 4: Already voted for candidate 248493ec312e4398b828c6cb00e5c4b1 in this term.
I20251026 14:59:05.683585 23934 consensus_queue.cc:260] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.770872 23934 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:05.718984 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:59:05.717981 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 79044.5us
I20251026 14:59:05.773588 22799 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:05.773900 22700 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:05.774597 24011 raft_consensus.cc:2804] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:05.775030 24011 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:05.775355 24011 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:05.775512 22700 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.777495 23845 consensus_queue.cc:1059] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:59:05.736795 22805 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:05.722699 22807 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:05.722559 24060 raft_consensus.cc:2804] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251026 14:59:05.779095 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 232806us
I20251026 14:59:05.779676 23845 raft_consensus.cc:3055] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:05.780200 23845 raft_consensus.cc:740] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:05.748088 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:05.728597 23929 consensus_queue.cc:1048] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20251026 14:59:05.781520 23959 raft_consensus.cc:2804] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:05.781962 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:05.727264 22774 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.782663 22802 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:05.743364 24042 raft_consensus.cc:2749] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:05.769631 23904 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:05.729113 22554 catalog_manager.cc:5649] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 1 to 2, leader changed from df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 2 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:05.784310 23904 raft_consensus.cc:3060] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:05.784463 22804 raft_consensus.cc:1240] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:05.781550 23740 raft_consensus.cc:2749] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:05.786202 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:05.785988 22554 catalog_manager.cc:5649] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 2 to 4, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 4 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:05.787735 22554 catalog_manager.cc:5649] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:05.782063 23959 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:05.789484 23959 raft_consensus.cc:3060] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:05.789719 24076 raft_consensus.cc:2764] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20251026 14:59:05.792066 23904 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:05.792537 24071 log.cc:927] Time spent T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20251026 14:59:05.793503 22700 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:05.793859 23898 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:05.718159 24044 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.794386 23898 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.795914 24091 raft_consensus.cc:2804] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:05.745977 23765 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:05.781211 23845 consensus_queue.cc:260] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.796826 23898 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:05.797842 23845 raft_consensus.cc:3060] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:05.794351 23904 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Requested vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:05.803567 22622 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:05.804899 23819 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:05.805666 23819 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.758592 24010 consensus_queue.cc:260] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.807919 23819 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:05.808668 24010 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:05.809911 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:05.810972 22651 raft_consensus.cc:2393] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:05.813922 24012 raft_consensus.cc:3055] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:05.814342 24012 raft_consensus.cc:740] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:05.814898 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:05.814988 24044 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:05.815141 24012 consensus_queue.cc:260] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:05.816556 24012 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:05.819483 23400 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 1 LEARNER]: Advancing to term 3
W20251026 14:59:05.820218 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 93029.8us
I20251026 14:59:05.820945 22774 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:05.822263 24073 consensus_queue.cc:1048] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:59:05.823348 24076 raft_consensus.cc:2749] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251026 14:59:05.793213 23696 consensus_queue.cc:260] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.796329 24091 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:05.824577 24091 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
W20251026 14:59:05.813664 23740 consensus_queue.cc:1175] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" responder_term: 3 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)" } } last_received_current_leader { term: 0 index: 0 } }
I20251026 14:59:05.763639 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:05.828042 22624 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:05.833338 22774 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:05.835605 22657 raft_consensus.cc:1275] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:59:05.840036 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:05.841583 22659 raft_consensus.cc:2410] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:05.841651 22624 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:05.848526 22624 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:05.850359 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:05.854280 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:05.855768 22730 raft_consensus.cc:1240] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 3. Ops: []
I20251026 14:59:05.855661 22624 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:05.855226 22731 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:05.862825 22624 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:05.863965 22624 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:05.866195 22734 raft_consensus.cc:1240] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251026 14:59:05.865160 22624 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:05.867192 22732 raft_consensus.cc:1275] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:05.839218 24030 raft_consensus.cc:3060] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:05.865919 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:05.839164 23740 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:05.872143 22725 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:05.872889 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:05.873528 22734 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:05.873373 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:05.874825 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:05.875125 22725 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:05.876336 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:05.875384 22734 raft_consensus.cc:2376] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3. Re-sending same reply.
I20251026 14:59:05.869041 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:05.878628 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:05.879074 22732 raft_consensus.cc:2468] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:05.847221 23696 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:05.879837 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:05.882331 22624 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:05.882568 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:05.882887 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:05.886132 22732 raft_consensus.cc:2468] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:05.886211 22733 raft_consensus.cc:2376] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3. Re-sending same reply.
I20251026 14:59:05.886909 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:05.883431 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:05.887465 22729 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:05.847110 23904 raft_consensus.cc:2804] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:05.877055 22725 raft_consensus.cc:2376] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3. Re-sending same reply.
I20251026 14:59:05.888146 22732 raft_consensus.cc:2376] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3. Re-sending same reply.
W20251026 14:59:05.888530 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:05.889045 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:05.893713 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:05.838902 23944 consensus_queue.cc:1048] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20251026 14:59:05.888568 23898 raft_consensus.cc:2804] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:05.870803 23911 raft_consensus.cc:2804] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:05.868006 24026 consensus_queue.cc:1048] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20251026 14:59:05.918910 22808 raft_consensus.cc:1275] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:59:05.899178 23400 raft_consensus.cc:1275] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 LEARNER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:59:05.868343 23839 raft_consensus.cc:2804] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:05.861330 24045 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:05.854418 24008 raft_consensus.cc:2804] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:05.862493 24084 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:05.907315 22733 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:05.912492 22801 raft_consensus.cc:1275] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:05.913040 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:05.913339 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:05.871513 23833 raft_consensus.cc:2804] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:05.868642 23916 raft_consensus.cc:2749] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:05.897352 23641 consensus_queue.cc:1059] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20251026 14:59:05.870306 24079 consensus_queue.cc:1048] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20251026 14:59:05.914681 22697 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:05.915607 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:05.925122 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:05.958608 22726 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:05.901587 23904 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:05.950373 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:05.911441 23855 raft_consensus.cc:2804] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:05.924240 23898 raft_consensus.cc:697] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:59:05.963989 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 81746.3us
I20251026 14:59:05.917243 23911 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:05.874316 24028 consensus_queue.cc:1048] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20251026 14:59:05.914667 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:05.978091 22624 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:05.963052 22803 raft_consensus.cc:2393] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:05.927886 23839 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:05.917243 22728 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:05.907567 24076 consensus_queue.cc:1048] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20251026 14:59:05.916260 23929 consensus_queue.cc:1048] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20251026 14:59:05.933631 24084 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:05.932866 24008 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:05.901751 23740 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:05.952672 23833 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:59:06.030898 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 116305us
I20251026 14:59:05.951834 22730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 587) took 1433 ms. Trace:
I20251026 14:59:06.032171 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:05.959646 22727 raft_consensus.cc:1275] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:06.000635 22733 raft_consensus.cc:1275] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:59:06.032001 22730 rpcz_store.cc:276] 1026 14:59:04.518781 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:04.951137 (+432356us) service_pool.cc:225] Handling call
1026 14:59:05.551156 (+600019us) spinlock_profiling.cc:243] Waited 977 us on lock 0x7b5800250598. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8ce074451 00007fd8ce074302 00007fd8d6f0dad9 00007fd8d6090d30 00007fd8d615984b 00007fd8d61598d1 00007fd8d611b580 00007fd8d6105ee9 00007fd8d6105992 00007fd8cf69fd34
1026 14:59:05.551355 (+ 199us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:05.856406 (+305051us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:05.951529 (+ 95123us) spinlock_profiling.cc:243] Waited 304 ms on lock 0x7b540040d8d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:05.951749 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":701716608}
I20251026 14:59:05.964169 23855 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:05.951278 24031 consensus_queue.cc:1048] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.046s
I20251026 14:59:05.985967 22624 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:05.964365 23904 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:06.024360 22773 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:06.045634 22773 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:06.036144 22730 raft_consensus.cc:1240] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251026 14:59:06.048445 22773 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:06.036902 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 108380us
I20251026 14:59:05.979334 23911 raft_consensus.cc:3060] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:06.034374 24084 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:06.041827 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 55960.4us
I20251026 14:59:06.057735 22624 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:05.979030 22802 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:06.029959 24063 raft_consensus.cc:2804] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:06.050863 22622 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:06.020387 24062 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:05.951813 24045 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.040506 23819 raft_consensus.cc:2749] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:06.029472 23839 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:06.035269 23833 raft_consensus.cc:3060] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:06.034138 24008 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:06.055827 22730 raft_consensus.cc:1275] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:59:06.068202 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:06.041742 23855 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:06.049557 23916 raft_consensus.cc:2804] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election won for term 3
W20251026 14:59:06.118767 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 59625.9us
I20251026 14:59:06.144487 22803 raft_consensus.cc:1275] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:59:06.064047 24079 raft_consensus.cc:2804] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:06.122272 22734 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:06.063000 23699 raft_consensus.cc:2764] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20251026 14:59:06.121697 24098 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:06.061926 23944 raft_consensus.cc:2764] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20251026 14:59:06.060683 24063 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:06.132251 22697 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:06.119102 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:06.136052 24089 log.cc:927] Time spent T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.153s user 0.000s sys 0.001s
I20251026 14:59:06.102259 23641 consensus_queue.cc:1059] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.078s
I20251026 14:59:06.073014 24099 raft_consensus.cc:2749] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:59:06.064293 23959 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.066108 24062 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.137310 22728 raft_consensus.cc:1275] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:59:06.123106 24076 consensus_queue.cc:1048] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.056s
I20251026 14:59:06.065503 23898 consensus_queue.cc:237] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.143996 23403 raft_consensus.cc:2468] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:06.072098 24087 consensus_queue.cc:1048] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.090s
W20251026 14:59:06.184985 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 134280us
I20251026 14:59:06.170250 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:06.179546 24063 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:06.177145 23934 raft_consensus.cc:2804] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:06.143007 24011 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.186056 23819 raft_consensus.cc:2749] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:06.231868 22622 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:06.181926 22799 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
W20251026 14:59:06.245543 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 100647us
I20251026 14:59:06.246366 22624 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:06.254446 22697 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:06.255982 24090 raft_consensus.cc:2749] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:06.218979 24060 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:06.185199 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 132681us
I20251026 14:59:06.236066 23934 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:06.235697 24070 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:06.261785 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:06.280432 22697 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:06.241381 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:06.267585 24060 raft_consensus.cc:3060] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:06.279506 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:06.288311 22800 raft_consensus.cc:2468] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:06.290560 22697 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:06.262890 22727 raft_consensus.cc:1240] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251026 14:59:06.302461 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:06.305315 22651 raft_consensus.cc:2468] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:06.302888 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:06.307915 22732 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:06.303478 22659 raft_consensus.cc:1275] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:59:06.264004 23740 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:06.297456 24079 raft_consensus.cc:697] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
W20251026 14:59:06.285732 24051 consensus_meta.cc:287] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.513s user 0.007s sys 0.003s
I20251026 14:59:06.313539 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:06.317021 22658 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:06.277710 23934 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
W20251026 14:59:06.311053 23765 consensus_meta.cc:287] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.514s user 0.010s sys 0.000s
I20251026 14:59:06.286944 24056 raft_consensus.cc:2749] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:06.239610 24086 consensus_queue.cc:1048] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.076s
W20251026 14:59:06.327033 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:06.304431 23819 raft_consensus.cc:2804] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:06.238054 24103 consensus_queue.cc:1048] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.090s
I20251026 14:59:06.329085 22726 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:06.299535 23916 raft_consensus.cc:697] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:06.338109 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:06.340145 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:06.346496 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 100248us
W20251026 14:59:06.335064 24091 consensus_meta.cc:287] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.510s user 0.009s sys 0.003s
W20251026 14:59:06.334532 24010 consensus_meta.cc:287] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.525s user 0.009s sys 0.001s
I20251026 14:59:06.341437 23819 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:06.286353 24100 consensus_queue.cc:1048] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.052s
I20251026 14:59:06.317982 24111 consensus_queue.cc:1048] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20251026 14:59:06.364392 22700 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:06.334266 24031 consensus_queue.cc:1048] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.022s
I20251026 14:59:06.344414 24045 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:06.301595 24070 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.334877 23765 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:06.368562 24096 log.cc:927] Time spent T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.246s user 0.000s sys 0.001s
I20251026 14:59:06.319001 24051 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.364207 23819 raft_consensus.cc:3060] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:06.359237 24108 raft_consensus.cc:2804] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:06.394445 22800 raft_consensus.cc:1275] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:59:06.380617 24090 raft_consensus.cc:2804] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:06.382406 24062 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:06.408159 22732 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:06.384096 23959 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:06.356271 24091 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.432121 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:06.434703 22732 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:06.436188 22773 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:06.430004 24108 raft_consensus.cc:697] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:06.431353 24099 raft_consensus.cc:493] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:06.431252 24112 consensus_queue.cc:1048] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.008s
I20251026 14:59:06.408756 24010 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:06.390864 23740 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:06.451290 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 52121.5us
I20251026 14:59:06.453819 22732 raft_consensus.cc:1240] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251026 14:59:06.403535 24090 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:06.374336 24081 consensus_queue.cc:1048] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
I20251026 14:59:06.334614 24079 consensus_queue.cc:237] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:59:06.405825 24011 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:06.463568 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 116046us
I20251026 14:59:06.429179 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:06.465693 22624 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:06.469069 22732 raft_consensus.cc:1240] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251026 14:59:06.473332 22622 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:06.480511 23839 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.482928 23839 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:06.486449 22624 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
W20251026 14:59:06.486989 24097 log.cc:927] Time spent T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.201s user 0.001s sys 0.000s
I20251026 14:59:06.458176 24115 raft_consensus.cc:2804] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:06.490097 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:06.460898 24090 raft_consensus.cc:3060] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:06.491631 22732 raft_consensus.cc:1275] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:59:06.494781 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:06.430205 23916 consensus_queue.cc:237] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.500242 22624 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:06.504060 22622 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:06.460374 23740 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.507440 22624 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:06.491017 24011 raft_consensus.cc:2804] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:06.484545 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:06.463482 24010 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.489795 24115 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:06.511523 22624 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:06.516448 22624 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:06.470868 23959 consensus_queue.cc:1048] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20251026 14:59:06.474473 24062 consensus_queue.cc:1059] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
W20251026 14:59:06.508373 24095 log.cc:927] Time spent T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.213s user 0.001s sys 0.000s
I20251026 14:59:06.455868 24099 raft_consensus.cc:515] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:06.508488 24093 log.cc:927] Time spent T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.234s user 0.001s sys 0.000s
I20251026 14:59:06.487576 23696 consensus_queue.cc:1059] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20251026 14:59:06.512708 24011 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:06.507184 24098 raft_consensus.cc:2749] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:06.522502 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:06.510249 24103 raft_consensus.cc:2804] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:06.523370 22655 raft_consensus.cc:3060] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:06.519256 24115 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:06.508759 23941 consensus_queue.cc:1048] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20251026 14:59:06.526032 24070 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:06.530376 24062 raft_consensus.cc:3055] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:06.583837 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:06.584682 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:06.528638 24120 raft_consensus.cc:2804] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:06.537068 23765 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:06.534782 23898 raft_consensus.cc:2749] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:06.590816 24117 raft_consensus.cc:2804] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:06.591266 24104 raft_consensus.cc:2804] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:06.593068 22697 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:06.532848 24011 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:06.596076 24122 raft_consensus.cc:2804] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251026 14:59:06.596248 24113 log.cc:927] Time spent T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.193s user 0.000s sys 0.001s
I20251026 14:59:06.509224 24108 consensus_queue.cc:237] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.591820 24120 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:06.598551 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:06.575148 24103 raft_consensus.cc:697] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:59:06.588685 23845 consensus_meta.cc:287] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.790s user 0.006s sys 0.006s
I20251026 14:59:06.582974 24051 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:06.607661 22657 raft_consensus.cc:1240] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20251026 14:59:06.607760 24104 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:06.585654 24062 raft_consensus.cc:740] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:06.535286 23904 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.598367 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:06.535347 23833 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.618197 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:06.607877 24117 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:06.606108 23765 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:06.609176 24091 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:06.611730 24122 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:06.629493 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:06.629154 24124 raft_consensus.cc:2804] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:06.627089 22697 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:06.638495 24121 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:06.639854 24123 log.cc:927] Time spent T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
W20251026 14:59:06.642510 23911 consensus_meta.cc:287] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.578s user 0.008s sys 0.003s
I20251026 14:59:06.642879 23833 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:06.616546 23898 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:06.643975 22659 raft_consensus.cc:2468] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:06.645802 22650 raft_consensus.cc:1240] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:06.619077 24120 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:06.628603 24104 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:06.654877 22652 raft_consensus.cc:2393] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:06.654803 22657 raft_consensus.cc:1275] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:59:06.645126 24117 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
W20251026 14:59:06.663543 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:06.666828 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:06.707213 22651 raft_consensus.cc:2393] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:06.667587 23898 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:06.649078 24122 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:06.665120 23911 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.671331 24010 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:06.673426 24062 consensus_queue.cc:260] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.665831 24091 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:06.651001 23740 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:06.646932 24051 raft_consensus.cc:2749] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20251026 14:59:06.664136 24084 consensus_meta.cc:287] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.593s user 0.006s sys 0.004s
W20251026 14:59:06.670703 24030 consensus_meta.cc:287] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.790s user 0.007s sys 0.003s
I20251026 14:59:06.712109 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:06.713253 23898 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
W20251026 14:59:06.696201 24012 consensus_meta.cc:287] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.879s user 0.010s sys 0.006s
I20251026 14:59:06.697321 24099 leader_election.cc:290] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:06.713676 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:06.740689 22726 raft_consensus.cc:2393] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:06.648988 23765 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.720932 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:06.660275 24124 raft_consensus.cc:697] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:06.745034 24125 raft_consensus.cc:2749] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251026 14:59:06.737061 24030 raft_consensus.cc:2749] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:06.759591 24026 raft_consensus.cc:3055] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:06.762981 24026 raft_consensus.cc:740] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
W20251026 14:59:06.764211 23855 consensus_meta.cc:287] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.643s user 0.007s sys 0.003s
I20251026 14:59:06.763980 24026 consensus_queue.cc:260] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.767333 22726 raft_consensus.cc:1275] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:59:06.761736 22771 leader_election.cc:304] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:06.764653 23855 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.749714 24062 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:06.769877 22658 raft_consensus.cc:1240] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:06.735543 24091 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.775123 24091 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:06.737423 24084 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.776378 22805 raft_consensus.cc:1240] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5. Ops: []
W20251026 14:59:06.756510 24131 log.cc:927] Time spent T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.103s user 0.001s sys 0.000s
I20251026 14:59:06.664891 24121 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:06.767316 24008 consensus_meta.cc:287] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.647s user 0.009s sys 0.002s
I20251026 14:59:06.756848 23898 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:06.789601 23904 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:06.777383 24084 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:06.777475 24026 raft_consensus.cc:3060] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:06.777961 23696 raft_consensus.cc:2749] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:06.771936 23833 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:06.807605 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 291247us
I20251026 14:59:06.714403 24103 consensus_queue.cc:237] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.756110 23819 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.825523 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:06.815608 24060 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:06.822001 24124 consensus_queue.cc:237] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:06.849613 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 222600us
I20251026 14:59:06.841895 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:06.853606 22725 raft_consensus.cc:2468] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:59:06.854218 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:06.841008 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 322356us
I20251026 14:59:06.854796 22725 raft_consensus.cc:1275] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:59:06.799151 24012 consensus_queue.cc:1048] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.042s
I20251026 14:59:06.862010 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:06.802031 24008 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.805637 23898 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.868862 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:06.869222 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:06.870036 22801 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:06.866861 22624 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:06.871778 24051 consensus_queue.cc:1048] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:59:06.873756 22655 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:06.878449 22700 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:06.873016 22624 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:06.810623 23934 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.882799 22651 raft_consensus.cc:1240] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251026 14:59:06.875164 23819 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:06.892966 22697 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:06.838675 23833 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:59:06.875151 23999 raft_consensus.cc:2804] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:06.902695 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:06.861443 24042 consensus_queue.cc:1048] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20251026 14:59:06.916122 22697 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:06.910171 22700 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:06.890851 24138 consensus_queue.cc:1048] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20251026 14:59:06.883299 23911 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:06.922792 23819 raft_consensus.cc:2804] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:06.923316 23819 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:06.923775 23819 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:06.923707 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:06.924237 24087 raft_consensus.cc:2749] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:06.875332 24098 consensus_queue.cc:1048] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.031s
I20251026 14:59:06.927223 24138 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:06.926844 24104 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.928117 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:06.929606 24138 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.930063 23934 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:06.933398 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:06.905025 24125 raft_consensus.cc:2804] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:06.904325 24051 raft_consensus.cc:2749] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:06.905978 23999 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:06.947095 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:06.947736 22803 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:06.948004 24138 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:06.950469 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:06.906492 24137 raft_consensus.cc:2804] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:06.953418 24137 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:06.953821 24137 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:06.956902 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:06.957600 22653 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:06.949776 24120 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:06.935726 23765 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:06.961704 22803 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:06.964272 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:06.956187 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:06.966452 22808 raft_consensus.cc:3060] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:06.969120 22697 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:06.975369 23765 raft_consensus.cc:2749] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:06.946256 23999 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:06.928738 24100 consensus_queue.cc:1059] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20251026 14:59:06.946532 24125 raft_consensus.cc:697] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:06.985858 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:06.961745 23855 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:06.978811 23765 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:06.995150 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:07.004045 24008 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.006565 22622 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:06.964177 23699 consensus_queue.cc:1048] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20251026 14:59:07.018575 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:07.018709 22774 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:07.029098 23898 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.033756 24100 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:07.034410 24100 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:07.036048 23833 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.005064 23765 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.049414 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:07.051954 22774 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:07.055027 23765 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.065644 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:07.066356 22653 raft_consensus.cc:3060] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:07.072580 22733 raft_consensus.cc:1240] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:07.016314 24098 raft_consensus.cc:2804] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
W20251026 14:59:07.061515 22802 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.026s sys 0.069s
W20251026 14:59:07.085498 22802 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.555s user 0.028s sys 0.069s
I20251026 14:59:07.085791 24104 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.085878 24120 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.082530 22733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1047) took 1206 ms. Trace:
I20251026 14:59:07.087862 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:07.087648 22733 rpcz_store.cc:276] 1026 14:59:05.876319 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:06.035701 (+159382us) service_pool.cc:225] Handling call
1026 14:59:06.153531 (+117830us) spinlock_profiling.cc:243] Waited 1.07 ms on lock 0x7b5800250598. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8ce074451 00007fd8ce074302 00007fd8d6f0dad9 00007fd8d6090d30 00007fd8d615984b 00007fd8d61598d1 00007fd8d611b580 00007fd8d6105ee9 00007fd8d6105992 00007fd8cf69fd34
1026 14:59:07.072203 (+918672us) spinlock_profiling.cc:243] Waited 2.43 ms on lock 0x7b5800250598. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8ce07477c 00007fd8ce07433b 00007fd8d6f0dad9 00007fd8d6090d30 00007fd8d615984b 00007fd8d61598d1 00007fd8d611b580 00007fd8d6105ee9 00007fd8d6105992 00007fd8cf69fd34
1026 14:59:07.072407 (+ 204us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:07.082146 (+ 9739us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:07.082443 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8049792}
I20251026 14:59:07.044749 24084 raft_consensus.cc:2804] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251026 14:59:07.068497 24063 consensus_meta.cc:287] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.813s user 0.013s sys 0.000s
W20251026 14:59:07.073220 24090 consensus_meta.cc:287] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.568s user 0.011s sys 0.000s
I20251026 14:59:07.091130 22801 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:59:07.042989 24100 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:07.061431 24086 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:07.097151 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:07.110517 22733 raft_consensus.cc:3060] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:07.119489 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:07.095114 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:07.120134 22803 raft_consensus.cc:2468] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:07.093457 24111 raft_consensus.cc:2804] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:07.120441 22799 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:07.120383 23699 consensus_queue.cc:1059] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20251026 14:59:07.113425 24121 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.097164 24084 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:07.109722 24098 raft_consensus.cc:697] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:07.130143 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:07.117878 24100 raft_consensus.cc:2749] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:07.136271 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:07.137914 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:07.138085 24073 raft_consensus.cc:3055] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:07.094345 24063 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:07.138623 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:07.139540 22805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 855) took 1009 ms. Trace:
I20251026 14:59:07.140326 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 632) took 1272 ms. Trace:
I20251026 14:59:07.160820 22734 rpcz_store.cc:276] 1026 14:59:05.872075 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:05.915409 (+ 43334us) service_pool.cc:225] Handling call
1026 14:59:07.140152 (+1224743us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b540008d4d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:07.140267 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":297948544}
I20251026 14:59:07.162348 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:07.133697 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:07.147833 22697 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:07.148061 22658 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:07.163218 22733 raft_consensus.cc:2468] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:07.151868 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:07.159970 22805 rpcz_store.cc:276] 1026 14:59:06.129858 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:06.130050 (+ 192us) service_pool.cc:225] Handling call
1026 14:59:06.130479 (+ 429us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:06.776803 (+646324us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:07.139288 (+362485us) spinlock_profiling.cc:243] Waited 646 ms on lock 0x7b54003354d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:07.139477 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1484786560}
I20251026 14:59:07.155653 24073 raft_consensus.cc:740] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:07.130239 24084 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:07.125465 24125 consensus_queue.cc:237] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.101378 24090 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.118773 24011 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.158634 24081 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:07.167498 23999 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.181432 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:07.197571 22651 raft_consensus.cc:1240] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:07.207237 22729 raft_consensus.cc:1240] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:07.219517 22622 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:07.245884 22624 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:07.173087 22808 raft_consensus.cc:2468] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:07.175453 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:07.255079 22697 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:07.256081 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:07.184715 22807 raft_consensus.cc:2468] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 4.
W20251026 14:59:07.230718 24122 consensus_meta.cc:287] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.507s user 0.006s sys 0.005s
I20251026 14:59:07.262295 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:07.174536 24115 consensus_meta.cc:287] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.589s user 0.010s sys 0.002s
I20251026 14:59:07.259788 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:07.263387 22651 raft_consensus.cc:2468] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 4.
I20251026 14:59:07.118628 24086 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.226323 22653 raft_consensus.cc:2468] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:07.273905 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:07.175678 23855 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:07.280915 22800 raft_consensus.cc:1275] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:07.302098 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:07.261821 24051 raft_consensus.cc:2804] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election won for term 4
W20251026 14:59:07.304814 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 212040us
I20251026 14:59:07.260202 22624 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:07.252519 24011 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.200331 23839 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:07.189083 24142 raft_consensus.cc:2804] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:07.272939 22653 raft_consensus.cc:1240] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:07.258267 24104 raft_consensus.cc:2804] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:07.324533 23839 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.213136 23941 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:07.325774 23941 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.326846 23839 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.327873 23941 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:07.329057 22653 raft_consensus.cc:1240] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:07.329360 22656 raft_consensus.cc:1240] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:07.330717 23941 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:07.313987 24103 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:07.331338 23941 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:07.332003 24103 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.334463 23941 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:59:07.318342 24100 consensus_queue.cc:1059] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20251026 14:59:07.334647 24103 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.335297 24100 raft_consensus.cc:3055] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:07.335661 24100 raft_consensus.cc:740] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:07.310261 23898 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:07.195053 24098 consensus_queue.cc:237] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.275090 22733 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:07.189204 24050 consensus_queue.cc:1059] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.092s
I20251026 14:59:07.320616 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:07.324288 24142 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:07.325042 24104 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:07.234688 23959 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:07.279668 22651 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "143c213a935041868ad3b0d9584ad863"
dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
mode: GRACEFUL
new_leader_uuid: "0d684445287b450a8f738b6ad4b59821"
from {username='slave'} at 127.0.0.1:46598
I20251026 14:59:07.330595 22653 raft_consensus.cc:1240] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:07.273846 24115 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.303081 23367 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters:
I20251026 14:59:07.297406 23855 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.273046 24122 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.460645 22808 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:07.396993 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:07.428778 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 63080.7us
I20251026 14:59:07.313652 24144 consensus_queue.cc:1059] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20251026 14:59:07.463320 22653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1054) took 1044 ms. Trace:
I20251026 14:59:07.468104 22653 rpcz_store.cc:276] 1026 14:59:06.418330 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:07.330011 (+911681us) service_pool.cc:225] Handling call
1026 14:59:07.330406 (+ 395us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:07.460558 (+130152us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:07.460828 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:07.463428 22804 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:07.464450 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:07.464910 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:07.473177 22733 raft_consensus.cc:2468] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:07.191622 24063 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:07.397732 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 137603us
W20251026 14:59:07.336146 24117 consensus_meta.cc:287] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.617s user 0.005s sys 0.007s
I20251026 14:59:07.174619 24073 consensus_queue.cc:260] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.476099 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:07.481069 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:07.485462 22656 raft_consensus.cc:1240] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:07.429929 24103 raft_consensus.cc:2804] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:07.487316 24063 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:07.418375 24087 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:07.488235 22655 raft_consensus.cc:1240] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251026 14:59:07.337746 24051 raft_consensus.cc:697] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:07.418974 23765 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:07.370224 23999 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.479638 24121 raft_consensus.cc:2804] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:07.466143 24104 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:07.206871 24081 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.461616 24032 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:07.428717 23934 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:07.491554 24144 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:07.322142 23833 consensus_queue.cc:1048] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.164s
I20251026 14:59:07.465880 24142 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:07.473397 24090 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.461089 24076 raft_consensus.cc:493] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:07.466923 23898 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:07.397256 24062 consensus_queue.cc:1059] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.118s
I20251026 14:59:07.499586 24103 raft_consensus.cc:697] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:07.503648 24060 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.504943 23699 raft_consensus.cc:3055] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:07.479031 24050 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:07.429689 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:07.510999 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:07.478029 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 204518us
I20251026 14:59:07.472563 24100 consensus_queue.cc:260] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.515391 24086 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.517315 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:07.577977 22805 raft_consensus.cc:2410] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:07.517809 22807 raft_consensus.cc:1240] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:07.471223 23959 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.511493 23765 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:07.398165 23941 consensus_queue.cc:1059] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.111s
I20251026 14:59:07.525213 22734 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:07.531662 24032 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.542778 22650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1043) took 1193 ms. Trace:
I20251026 14:59:07.608255 22650 rpcz_store.cc:276] 1026 14:59:06.349314 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:06.501610 (+152296us) service_pool.cc:225] Handling call
1026 14:59:06.510998 (+ 9388us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:06.675529 (+164531us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:07.529797 (+854268us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b54000dc854. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:07.542696 (+ 12899us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":304009088}
I20251026 14:59:07.543897 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1036) took 1208 ms. Trace:
I20251026 14:59:07.609627 22659 rpcz_store.cc:276] 1026 14:59:06.335669 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:06.337796 (+ 2127us) service_pool.cc:225] Handling call
1026 14:59:07.529453 (+1191657us) spinlock_profiling.cc:243] Waited 283 ms on lock 0x7b540012e8d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:07.543820 (+ 14367us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":650642688}
I20251026 14:59:07.510675 24121 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:07.551996 24081 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.490420 24117 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.496569 24063 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.562481 22808 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:07.620391 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:07.369064 24012 consensus_queue.cc:1059] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.196s
I20251026 14:59:07.559626 24090 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:07.566300 23699 raft_consensus.cc:740] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:07.566748 24099 raft_consensus.cc:3055] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Stepping down as leader of term 2
W20251026 14:59:07.562377 24096 log.cc:927] Time spent T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 1.062s user 0.002s sys 0.005s
I20251026 14:59:07.544845 23934 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.480931 22624 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:07.607177 22771 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.010s sys 0.023s
I20251026 14:59:07.621196 22659 raft_consensus.cc:2393] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:07.623049 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:07.516562 24087 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:07.583621 24086 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:07.621033 24121 raft_consensus.cc:3060] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:07.633107 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 152352us
I20251026 14:59:07.657135 22624 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:07.626245 24099 raft_consensus.cc:740] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:07.626307 24081 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:07.572561 24050 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:07.644690 24026 consensus_meta.cc:287] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.838s user 0.011s sys 0.002s
I20251026 14:59:07.634110 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:59:07.663177 24137 consensus_meta.cc:287] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.709s user 0.011s sys 0.003s
I20251026 14:59:07.666864 22733 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:59:07.644203 23833 raft_consensus.cc:2749] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:07.646154 22651 raft_consensus.cc:606] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Received request to transfer leadership to TS 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:07.671229 22651 raft_consensus.cc:614] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
W20251026 14:59:07.633009 22725 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.701s user 0.017s sys 0.021s
W20251026 14:59:07.671880 22725 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.740s user 0.017s sys 0.021s
I20251026 14:59:07.654759 22653 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
W20251026 14:59:07.632810 22771 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.563s user 0.010s sys 0.023s
I20251026 14:59:07.623963 24062 raft_consensus.cc:3055] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:07.658279 22624 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:07.565999 24076 raft_consensus.cc:515] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:07.556640 24144 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.634532 22808 raft_consensus.cc:3060] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:07.645421 22734 raft_consensus.cc:2468] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:07.567375 24051 consensus_queue.cc:237] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:07.632279 24114 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.502s user 0.021s sys 0.074s
W20251026 14:59:07.646232 22732 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.758s user 0.030s sys 0.078s
I20251026 14:59:07.659312 23904 raft_consensus.cc:2749] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251026 14:59:07.661829 24012 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:07.661713 23855 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.629900 24090 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.661129 24141 raft_consensus.cc:2804] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251026 14:59:07.679597 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:07.687842 23934 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.672391 23699 consensus_queue.cc:260] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.672081 24160 raft_consensus.cc:3055] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:07.693439 22730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1061) took 1501 ms. Trace:
I20251026 14:59:07.702668 24011 raft_consensus.cc:2749] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20251026 14:59:07.706295 22732 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.818s user 0.030s sys 0.078s
I20251026 14:59:07.729539 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1059) took 1667 ms. Trace:
I20251026 14:59:07.645514 24103 consensus_queue.cc:237] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.662140 24086 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.717831 24076 leader_election.cc:290] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:07.666352 24081 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.684619 24122 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.668567 24137 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:07.755610 24149 log.cc:927] Time spent T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.166s user 0.001s sys 0.000s
W20251026 14:59:07.733769 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:07.756857 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:07.731374 22730 rpcz_store.cc:276] 1026 14:59:06.192173 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:06.755441 (+563268us) service_pool.cc:225] Handling call
1026 14:59:06.755776 (+ 335us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:06.758654 (+ 2878us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:06.758686 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:06.779179 (+ 20493us) log.cc:844] Serialized 26 byte log entry
1026 14:59:07.262491 (+483312us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:07.262539 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:07.262602 (+ 63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:07.693198 (+430596us) raft_consensus.cc:1685] finished
1026 14:59:07.693218 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:07.693382 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2504,"raft.queue_time_us":2861,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":17704,"thread_start_us":433825,"threads_started":1,"wal-append.queue_time_us":826713}
I20251026 14:59:07.730057 22732 rpcz_store.cc:276] 1026 14:59:06.062462 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:06.493739 (+431277us) service_pool.cc:225] Handling call
1026 14:59:06.509145 (+ 15406us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:06.598444 (+ 89299us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:06.598478 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:06.605151 (+ 6673us) log.cc:844] Serialized 26 byte log entry
1026 14:59:06.864570 (+259419us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:07.728776 (+864206us) log.cc:844] Serialized 14 byte log entry
1026 14:59:07.729081 (+ 305us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:07.729180 (+ 99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:07.729240 (+ 60us) raft_consensus.cc:1685] finished
1026 14:59:07.729259 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:07.729463 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23390,"prepare.queue_time_us":487092,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":1364,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":11281,"thread_start_us":1164520,"threads_started":3,"wal-append.queue_time_us":439879}
I20251026 14:59:07.693867 24062 raft_consensus.cc:740] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:07.730700 24160 raft_consensus.cc:740] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:07.753548 23699 raft_consensus.cc:3060] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:07.754739 24086 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.754623 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:07.755299 24081 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.708910 24142 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.761358 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:07.773084 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:07.765916 22771 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:07.773952 22732 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:07.767230 24137 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.739359 24056 raft_consensus.cc:1064] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e: attempting to promote NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 to VOTER
I20251026 14:59:07.777710 24111 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:59:07.777925 23819 consensus_meta.cc:287] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.853s user 0.013s sys 0.000s
W20251026 14:59:07.777956 24056 raft_consensus.cc:1068] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e: Unable to promote non-voter 248493ec312e4398b828c6cb00e5c4b1: Illegal state: Leader has not yet committed an operation in its own term
I20251026 14:59:07.776247 23765 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:07.689426 24099 consensus_queue.cc:260] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.778734 24090 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.779747 22728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1054) took 1871 ms. Trace:
I20251026 14:59:07.776654 24023 raft_consensus.cc:2749] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:07.752339 23959 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.780201 22728 rpcz_store.cc:276] 1026 14:59:05.908048 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:06.265698 (+357650us) service_pool.cc:225] Handling call
1026 14:59:06.333222 (+ 67524us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:06.363999 (+ 30777us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:06.369877 (+ 5878us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:06.464962 (+ 95085us) log.cc:844] Serialized 52 byte log entry
1026 14:59:06.722052 (+257090us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:06.722107 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:07.779291 (+1057184us) spinlock_profiling.cc:243] Waited 6.11 ms on lock 0x7b540008ded4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:07.779343 (+ 52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:07.779398 (+ 55us) raft_consensus.cc:1685] finished
1026 14:59:07.779414 (+ 16us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:07.779674 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4584,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":13516,"spinlock_wait_cycles":14062976,"thread_start_us":209357,"threads_started":1,"wal-append.queue_time_us":359000}
I20251026 14:59:07.782580 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:07.782661 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:07.783089 24099 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:07.783274 22728 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:07.778205 24111 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:07.778416 23819 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.721658 24012 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.735752 23898 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:07.773527 22730 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:07.790252 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:07.787453 22808 raft_consensus.cc:2468] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 4.
I20251026 14:59:07.788338 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:07.776705 24032 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.738550 22802 raft_consensus.cc:2468] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:07.791013 22734 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:07.792855 23819 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.794785 24063 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.771795 24115 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:07.776757 24050 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:07.812877 22734 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:07.793202 24117 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.808864 24087 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:07.828568 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:07.801663 24114 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:07.841306 22732 raft_consensus.cc:2468] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:07.842658 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:07.843391 22732 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
W20251026 14:59:07.845394 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:07.846431 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:07.801057 23367 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters:
I20251026 14:59:07.798748 24062 consensus_queue.cc:260] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:07.871919 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:07.878705 22734 raft_consensus.cc:2393] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 4: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:07.811805 24144 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:07.883059 24109 raft_consensus.cc:3055] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:07.889372 23917 raft_consensus.cc:2804] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:07.892231 23917 raft_consensus.cc:697] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 4 LEADER]: Becoming Leader. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:07.894397 23917 consensus_queue.cc:237] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:07.896811 22655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1048) took 1546 ms. Trace:
I20251026 14:59:07.897307 22655 rpcz_store.cc:276] 1026 14:59:06.350575 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:06.874746 (+524171us) service_pool.cc:225] Handling call
1026 14:59:06.875130 (+ 384us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:07.496373 (+621243us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:07.896501 (+400128us) spinlock_profiling.cc:243] Waited 613 ms on lock 0x7b540009d8d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:07.896744 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1409224960}
I20251026 14:59:07.898566 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:07.900022 24142 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:07.902097 24073 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:07.845952 24114 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:07.885309 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 84312.9us
W20251026 14:59:07.902984 24030 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.593s user 0.013s sys 0.029s
I20251026 14:59:07.903275 22727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1051) took 2026 ms. Trace:
W20251026 14:59:07.927788 24044 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.651s user 0.012s sys 0.035s
I20251026 14:59:07.939535 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 645) took 1039 ms. Trace:
I20251026 14:59:07.913489 24109 raft_consensus.cc:740] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:07.958550 22734 rpcz_store.cc:276] 1026 14:59:06.899719 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:07.871537 (+971818us) service_pool.cc:225] Handling call
1026 14:59:07.939446 (+ 67909us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:07.966001 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 857) took 1441 ms. Trace:
I20251026 14:59:07.966535 22652 rpcz_store.cc:276] 1026 14:59:06.524311 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:06.581924 (+ 57613us) service_pool.cc:225] Handling call
1026 14:59:07.965662 (+1383738us) spinlock_profiling.cc:243] Waited 36.8 ms on lock 0x7b54001d74d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:07.965914 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":84607232}
W20251026 14:59:07.972518 22726 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.036s sys 0.020s
W20251026 14:59:07.973085 22726 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.710s user 0.036s sys 0.021s
I20251026 14:59:07.932246 22655 raft_consensus.cc:1240] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:07.976001 22657 raft_consensus.cc:1240] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251026 14:59:07.928452 24012 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:07.951023 24030 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.647s user 0.013s sys 0.029s
I20251026 14:59:07.980787 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47996 (request call id 72) took 1388 ms. Trace:
I20251026 14:59:07.982221 22734 rpcz_store.cc:276] 1026 14:59:06.592585 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:07.979721 (+1387136us) service_pool.cc:225] Handling call
1026 14:59:07.980133 (+ 412us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:07.980350 (+ 217us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:07.980390 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:07.980448 (+ 58us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:07.980491 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:07.980564 (+ 73us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:07.980716 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:08.006318 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1063) took 1682 ms. Trace:
I20251026 14:59:07.949649 24076 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:07.949426 24144 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:07.961723 24044 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s user 0.013s sys 0.035s
I20251026 14:59:08.016043 22652 raft_consensus.cc:1240] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:08.016721 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1059) took 1080 ms. Trace:
I20251026 14:59:08.017050 22652 rpcz_store.cc:276] 1026 14:59:06.939111 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:07.979573 (+1040462us) service_pool.cc:225] Handling call
1026 14:59:08.015870 (+ 36297us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:08.016493 (+ 623us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:08.016666 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:07.981511 22799 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s user 0.011s sys 0.023s
I20251026 14:59:08.052476 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 866) took 1113 ms. Trace:
I20251026 14:59:07.979207 24142 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:08.053097 22652 rpcz_store.cc:276] 1026 14:59:06.939299 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:08.017858 (+1078559us) service_pool.cc:225] Handling call
1026 14:59:08.051580 (+ 33722us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:08.051843 (+ 263us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:08.051881 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:08.051943 (+ 62us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:08.051986 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:08.052059 (+ 73us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:08.052392 (+ 333us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:08.007128 22726 rpcz_store.cc:276] 1026 14:59:06.324034 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:06.828209 (+504175us) service_pool.cc:225] Handling call
1026 14:59:06.846389 (+ 18180us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:06.847007 (+ 618us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:06.847056 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:06.860051 (+ 12995us) log.cc:844] Serialized 52 byte log entry
1026 14:59:07.220025 (+359974us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:07.973557 (+753532us) log.cc:844] Serialized 14 byte log entry
1026 14:59:07.997762 (+ 24205us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:08.000860 (+ 3098us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:08.001096 (+ 236us) raft_consensus.cc:1685] finished
1026 14:59:08.001115 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:08.006230 (+ 5115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8,"prepare.queue_time_us":242839,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":36,"raft.queue_time_us":2438,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":9719,"thread_start_us":1037019,"threads_started":2,"wal-append.queue_time_us":500958}
I20251026 14:59:08.184567 24114 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:08.186698 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47996 (request call id 73) took 1572 ms. Trace:
I20251026 14:59:08.187290 22734 rpcz_store.cc:276] 1026 14:59:06.614516 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:07.988318 (+1373802us) service_pool.cc:225] Handling call
1026 14:59:08.186016 (+197698us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:08.186272 (+ 256us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:08.186309 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:08.186370 (+ 61us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:08.186413 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:08.186487 (+ 74us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:08.186625 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:08.191577 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:08.054893 22799 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s user 0.011s sys 0.024s
I20251026 14:59:08.216545 22732 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:08.218325 22732 raft_consensus.cc:1240] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251026 14:59:08.219105 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1071) took 1476 ms. Trace:
I20251026 14:59:08.219493 22732 rpcz_store.cc:276] 1026 14:59:06.742363 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:08.217592 (+1475229us) service_pool.cc:225] Handling call
1026 14:59:08.218032 (+ 440us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:08.218784 (+ 752us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:08.219037 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:08.222899 22730 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:08.223796 22730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 647) took 1020 ms. Trace:
I20251026 14:59:08.224203 22730 rpcz_store.cc:276] 1026 14:59:07.203014 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:07.760987 (+557973us) service_pool.cc:225] Handling call
1026 14:59:08.223722 (+462735us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:08.228559 22726 raft_consensus.cc:1240] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251026 14:59:08.229422 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1068) took 1601 ms. Trace:
I20251026 14:59:08.229817 22726 rpcz_store.cc:276] 1026 14:59:06.628128 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:08.185585 (+1557457us) service_pool.cc:225] Handling call
1026 14:59:08.228258 (+ 42673us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:08.229062 (+ 804us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:08.229352 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:08.232126 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:08.233110 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:08.191298 24152 log.cc:927] Time spent T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.177s user 0.000s sys 0.000s
I20251026 14:59:08.264348 22734 raft_consensus.cc:1240] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251026 14:59:08.277964 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1070) took 1561 ms. Trace:
I20251026 14:59:08.278462 22734 rpcz_store.cc:276] 1026 14:59:06.716552 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:08.188531 (+1471979us) service_pool.cc:225] Handling call
1026 14:59:08.264011 (+ 75480us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:08.277597 (+ 13586us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:08.277890 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:08.202507 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 302497us
I20251026 14:59:08.020198 24144 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.190163 24142 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.267454 23941 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:08.294215 23941 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.269443 24012 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:08.272146 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47996 (request call id 74) took 1402 ms. Trace:
I20251026 14:59:08.202456 24064 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:08.296346 22726 rpcz_store.cc:276] 1026 14:59:06.869891 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:08.230731 (+1360840us) service_pool.cc:225] Handling call
1026 14:59:08.271434 (+ 40703us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:08.271667 (+ 233us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:08.271731 (+ 64us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:08.271789 (+ 58us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:08.271834 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:08.271905 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:08.272056 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:08.298085 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1066) took 1792 ms. Trace:
I20251026 14:59:08.213586 24109 consensus_queue.cc:260] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.018997 24076 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.177088 22655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1058) took 1146 ms. Trace:
I20251026 14:59:07.955447 22727 rpcz_store.cc:276] 1026 14:59:05.877293 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:06.076010 (+198717us) service_pool.cc:225] Handling call
1026 14:59:06.170536 (+ 94526us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:06.314392 (+143856us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:07.880201 (+1565809us) spinlock_profiling.cc:243] Waited 49.4 ms on lock 0x7b5400077454. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:07.893353 (+ 13152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":113604352}
I20251026 14:59:08.231647 24115 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:08.298643 22725 rpcz_store.cc:276] 1026 14:59:06.505313 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:06.855501 (+350188us) service_pool.cc:225] Handling call
1026 14:59:06.855906 (+ 405us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:06.863246 (+ 7340us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:06.863286 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:06.881901 (+ 18615us) log.cc:844] Serialized 52 byte log entry
1026 14:59:07.672278 (+790377us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:07.672324 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:07.672403 (+ 79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:08.290980 (+618577us) raft_consensus.cc:1685] finished
1026 14:59:08.291007 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:08.298000 (+ 6993us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7010,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":11210,"thread_start_us":740760,"threads_started":1,"wal-append.queue_time_us":1106613}
I20251026 14:59:08.317761 22725 raft_consensus.cc:1275] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20251026 14:59:08.321225 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:08.302052 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:08.324337 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 644) took 1425 ms. Trace:
I20251026 14:59:08.300529 24050 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:08.329332 22725 rpcz_store.cc:276] 1026 14:59:06.899403 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:08.311629 (+1412226us) service_pool.cc:225] Handling call
1026 14:59:08.312004 (+ 375us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:08.319528 (+ 7524us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:08.324253 (+ 4725us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:08.334308 24162 log.cc:927] Time spent T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.156s user 0.000s sys 0.000s
I20251026 14:59:08.336453 22658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1049) took 1982 ms. Trace:
I20251026 14:59:08.336874 22658 rpcz_store.cc:276] 1026 14:59:06.354075 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:07.176323 (+822248us) service_pool.cc:225] Handling call
1026 14:59:07.197616 (+ 21293us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:07.198022 (+ 406us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:07.198051 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:07.210591 (+ 12540us) log.cc:844] Serialized 26 byte log entry
1026 14:59:07.633541 (+422950us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:07.633606 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:07.633714 (+ 108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:08.336225 (+702511us) raft_consensus.cc:1685] finished
1026 14:59:08.336245 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:08.336394 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":141,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":9628,"thread_start_us":398755,"threads_started":1,"wal-append.queue_time_us":1104816}
I20251026 14:59:08.312583 24109 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:08.307482 24064 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.362210 22658 raft_consensus.cc:1240] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251026 14:59:08.362998 22658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1062) took 1247 ms. Trace:
I20251026 14:59:08.363440 22658 rpcz_store.cc:276] 1026 14:59:07.115283 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:08.338494 (+1223211us) service_pool.cc:225] Handling call
1026 14:59:08.361941 (+ 23447us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:08.362662 (+ 721us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:08.362914 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:08.303884 24012 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:08.357563 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:08.367054 22654 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.807s user 0.000s sys 0.032s
I20251026 14:59:08.314559 24115 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.306720 22655 rpcz_store.cc:276] 1026 14:59:06.936064 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:07.899604 (+963540us) service_pool.cc:225] Handling call
1026 14:59:07.914567 (+ 14963us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:07.985105 (+ 70538us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:08.082555 (+ 97450us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:08.367372 22654 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.808s user 0.000s sys 0.032s
W20251026 14:59:08.367497 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:08.361235 23699 consensus_meta.cc:287] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.590s user 0.016s sys 0.002s
W20251026 14:59:08.349917 22699 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.892s user 0.014s sys 0.017s
I20251026 14:59:08.345057 24050 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.385159 22727 raft_consensus.cc:1240] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: [2.2-3.3]
W20251026 14:59:08.387696 22699 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.958s user 0.015s sys 0.017s
W20251026 14:59:08.392243 24104 consensus_meta.cc:287] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.869s user 0.007s sys 0.012s
W20251026 14:59:08.423246 22623 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.954s user 0.031s sys 0.042s
W20251026 14:59:08.436913 22623 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.968s user 0.031s sys 0.042s
W20251026 14:59:08.437755 22806 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.780s user 0.000s sys 0.032s
W20251026 14:59:08.438124 22806 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.784s user 0.000s sys 0.032s
W20251026 14:59:08.438560 22700 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.064s user 0.012s sys 0.044s
W20251026 14:59:08.438886 22700 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.064s user 0.012s sys 0.044s
W20251026 14:59:08.439179 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:08.441187 22622 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.970s user 0.014s sys 0.020s
W20251026 14:59:08.447559 22622 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.979s user 0.014s sys 0.020s
W20251026 14:59:08.399935 24084 consensus_meta.cc:287] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 1.200s user 0.002s sys 0.020s
W20251026 14:59:08.479504 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:08.456826 22698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.962s user 0.005s sys 0.029s
W20251026 14:59:08.492672 22698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.999s user 0.005s sys 0.029s
I20251026 14:59:08.482141 22806 raft_consensus.cc:2955] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
W20251026 14:59:08.489792 24079 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.823s user 0.008s sys 0.045s
W20251026 14:59:08.520743 22659 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.859s user 0.027s sys 0.028s
I20251026 14:59:08.455243 24087 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:08.485844 23951 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.996s user 0.020s sys 0.033s
W20251026 14:59:08.537955 22659 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.877s user 0.027s sys 0.028s
I20251026 14:59:08.491717 23941 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:08.481711 24030 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:59:08.541900 23951 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.053s user 0.025s sys 0.033s
W20251026 14:59:08.520599 24079 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.856s user 0.009s sys 0.046s
I20251026 14:59:08.455014 24104 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.499732 24142 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:08.502218 24144 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:08.517922 22727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1075) took 1559 ms. Trace:
I20251026 14:59:08.573642 24064 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:08.575074 24160 consensus_queue.cc:260] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.573580 22727 rpcz_store.cc:276] 1026 14:59:06.958813 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:08.315899 (+1357086us) service_pool.cc:225] Handling call
1026 14:59:08.352272 (+ 36373us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:08.435283 (+ 83011us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:08.500802 (+ 65519us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:08.576545 24160 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:08.576953 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:08.577731 22727 raft_consensus.cc:2468] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:59:08.578115 24115 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:08.578747 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:08.495888 24084 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.579756 22727 raft_consensus.cc:2393] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:08.580713 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:08.581321 22727 raft_consensus.cc:2468] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:08.581727 24171 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:08.582293 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:08.582362 24171 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.583197 22727 raft_consensus.cc:2393] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:08.583815 24142 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:08.584030 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:08.584316 24142 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.587185 24142 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:08.587721 24142 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:08.588168 24142 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:08.536536 24108 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.717s user 0.014s sys 0.003s
W20251026 14:59:08.590018 24108 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.771s user 0.015s sys 0.003s
I20251026 14:59:08.590324 24044 raft_consensus.cc:3055] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:08.590698 24044 raft_consensus.cc:740] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:08.591101 24142 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:08.591616 24084 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:08.591477 24044 consensus_queue.cc:260] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.592995 24076 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:08.558451 24030 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:08.594305 24146 log.cc:927] Time spent T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 1.280s user 0.005s sys 0.017s
I20251026 14:59:08.546756 24087 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.598042 24044 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:08.599916 22700 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:08.600150 24087 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:08.600805 24167 log.cc:927] Time spent T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.102s user 0.001s sys 0.002s
I20251026 14:59:08.549925 22655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32888 (request call id 42) took 1345 ms. Trace:
I20251026 14:59:08.601645 22655 rpcz_store.cc:276] 1026 14:59:07.200191 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:08.386615 (+1186424us) service_pool.cc:225] Handling call
1026 14:59:08.418214 (+ 31599us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:08.455152 (+ 36938us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:08.482508 (+ 27356us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:08.496377 (+ 13869us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:08.499312 (+ 2935us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:08.522811 (+ 23499us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:08.545660 (+ 22849us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:08.603485 24063 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:08.603982 24063 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:08.605168 24121 consensus_meta.cc:287] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1: Time spent flushing consensus metadata: real 0.940s user 0.012s sys 0.000s
I20251026 14:59:08.605713 24121 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:08.606389 24063 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:08.608181 24121 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:08.608208 22728 consensus_meta.cc:287] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.816s user 0.014s sys 0.005s
I20251026 14:59:08.609898 22728 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:08.610973 22728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 650) took 1403 ms. Trace:
I20251026 14:59:08.611238 24104 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:08.517207 23740 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:08.601159 22700 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:08.626606 22624 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:08.600948 24087 raft_consensus.cc:2749] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251026 14:59:08.584614 22727 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
W20251026 14:59:08.623055 24103 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.828s user 0.005s sys 0.018s
I20251026 14:59:08.661412 24084 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:08.649183 24168 log.cc:927] Time spent T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.063s user 0.000s sys 0.001s
I20251026 14:59:08.666767 22624 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
W20251026 14:59:08.676769 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:08.677318 22624 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:08.678423 24063 raft_consensus.cc:2804] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:08.678954 24063 raft_consensus.cc:697] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:08.680495 22624 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:08.648783 24104 raft_consensus.cc:2804] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:08.680012 24063 consensus_queue.cc:237] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.681416 24084 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:08.658103 24124 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.730s user 0.007s sys 0.024s
W20251026 14:59:08.725953 24124 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.813s user 0.007s sys 0.024s
I20251026 14:59:08.726794 24124 raft_consensus.cc:2804] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:08.727236 24084 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:08.683946 22624 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:08.728379 22806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 869) took 1669 ms. Trace:
W20251026 14:59:08.730434 22650 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.973s user 0.011s sys 0.064s
I20251026 14:59:08.727295 24124 raft_consensus.cc:697] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:59:08.731181 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 47362.7us
W20251026 14:59:08.731736 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 539378us
I20251026 14:59:08.731565 24124 consensus_queue.cc:237] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.732266 22624 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:08.733561 22624 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:08.733870 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:08.734776 22624 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:08.735499 24188 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:08.735651 22624 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:08.728899 22806 rpcz_store.cc:276] 1026 14:59:07.059385 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:07.072082 (+ 12697us) service_pool.cc:225] Handling call
1026 14:59:07.072628 (+ 546us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:07.572665 (+500037us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:07.572695 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:07.606657 (+ 33962us) log.cc:844] Serialized 511 byte log entry
1026 14:59:08.481945 (+875288us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:08.638370 (+156425us) log.cc:844] Serialized 14 byte log entry
1026 14:59:08.647947 (+ 9577us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:08.648062 (+ 115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:08.728049 (+ 79987us) raft_consensus.cc:1685] finished
1026 14:59:08.728089 (+ 40us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:08.728294 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9171,"raft.queue_time_us":56216,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":11028,"thread_start_us":1283781,"threads_started":2,"wal-append.queue_time_us":1097214}
I20251026 14:59:08.736061 24188 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.646764 24171 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:08.624908 22728 rpcz_store.cc:276] 1026 14:59:07.207751 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:07.782235 (+574484us) service_pool.cc:225] Handling call
1026 14:59:08.610895 (+828660us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:08.663249 24103 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.869s user 0.006s sys 0.018s
W20251026 14:59:08.647532 24125 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.125s user 0.004s sys 0.020s
W20251026 14:59:08.662271 24051 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.862s user 0.000s sys 0.023s
I20251026 14:59:08.646082 23740 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:08.745054 22650 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.987s user 0.011s sys 0.064s
I20251026 14:59:08.744967 22773 leader_election.cc:304] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:08.664062 24012 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:08.645944 24142 raft_consensus.cc:2749] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251026 14:59:08.754276 22808 raft_consensus.cc:1275] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20251026 14:59:08.757985 22774 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.214s user 0.017s sys 0.037s
I20251026 14:59:08.759384 22730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1074) took 1921 ms. Trace:
I20251026 14:59:08.737810 24174 raft_consensus.cc:2749] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20251026 14:59:08.763248 22774 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.220s user 0.017s sys 0.037s
I20251026 14:59:08.721961 24104 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:08.763509 22730 rpcz_store.cc:276] 1026 14:59:06.838309 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:08.225185 (+1386876us) service_pool.cc:225] Handling call
1026 14:59:08.225595 (+ 410us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:08.225844 (+ 249us) raft_consensus.cc:1534] Early marking committed up to index 2
1026 14:59:08.284902 (+ 59058us) log.cc:844] Serialized 14 byte log entry
1026 14:59:08.759081 (+474179us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:08.759131 (+ 50us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:08.759170 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:08.759233 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:08.759334 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17885,"prepare.queue_time_us":140301,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":18104,"thread_start_us":455583,"threads_started":1}
W20251026 14:59:08.774720 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 1.24024e+06us
W20251026 14:59:08.757792 24085 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.238s user 0.014s sys 0.040s
I20251026 14:59:08.699496 23898 raft_consensus.cc:2804] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:08.776225 22773 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:08.758430 23369 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.015s sys 0.019s
I20251026 14:59:08.741598 24173 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:08.804617 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:08.794827 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:08.805658 22801 raft_consensus.cc:3060] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:08.797122 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:59:08.800575 22652 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.017s sys 0.013s
W20251026 14:59:08.806763 22652 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.526s user 0.017s sys 0.013s
I20251026 14:59:08.807193 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:08.762655 24171 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:08.759114 23951 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:08.750809 24051 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.956s user 0.000s sys 0.023s
I20251026 14:59:08.824777 23951 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.825125 24051 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:08.825722 24051 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.826812 23951 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:08.804486 24174 raft_consensus.cc:2804] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:08.827689 24174 raft_consensus.cc:697] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:08.827823 24051 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:08.802425 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:08.803730 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:08.828653 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:08.829440 22803 raft_consensus.cc:2468] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
W20251026 14:59:08.804814 23369 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.616s user 0.016s sys 0.019s
W20251026 14:59:08.830346 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 810144us
I20251026 14:59:08.830633 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:08.751716 24050 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:08.805510 22808 raft_consensus.cc:2468] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:08.807992 22804 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:59:08.800621 23699 raft_consensus.cc:2804] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:08.812129 23898 raft_consensus.cc:697] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:08.797334 24114 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:08.814772 23740 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:08.815339 24169 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:08.812568 24172 log.cc:927] Time spent T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20251026 14:59:08.862237 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:59:08.819200 24073 consensus_meta.cc:287] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.917s user 0.007s sys 0.004s
I20251026 14:59:08.798476 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:08.775884 24170 raft_consensus.cc:2749] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:08.822539 24087 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:59:08.824312 24111 consensus_meta.cc:287] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 1.031s user 0.016s sys 0.004s
I20251026 14:59:08.782840 24103 raft_consensus.cc:2749] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20251026 14:59:08.750125 24125 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.227s user 0.004s sys 0.021s
I20251026 14:59:08.833595 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:08.793566 24104 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
W20251026 14:59:08.846516 22726 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.500s user 0.017s sys 0.012s
I20251026 14:59:08.849762 23951 raft_consensus.cc:2804] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:08.851637 23696 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
W20251026 14:59:08.799872 24085 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.280s user 0.014s sys 0.041s
I20251026 14:59:08.828636 24174 consensus_queue.cc:237] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:08.848435 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1073) took 2038 ms. Trace:
I20251026 14:59:08.855165 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:08.983112 22805 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:08.848066 23699 raft_consensus.cc:493] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
W20251026 14:59:08.865643 23917 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.514s user 0.028s sys 0.024s
I20251026 14:59:08.852633 22799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 871) took 1651 ms. Trace:
I20251026 14:59:08.868693 24142 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:08.848093 24171 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:08.951270 22726 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.605s user 0.024s sys 0.020s
I20251026 14:59:08.896334 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 861) took 2193 ms. Trace:
W20251026 14:59:08.941223 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:08.982831 22732 rpcz_store.cc:276] 1026 14:59:06.809798 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:08.220403 (+1410605us) service_pool.cc:225] Handling call
1026 14:59:08.267034 (+ 46631us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:08.267505 (+ 471us) raft_consensus.cc:1534] Early marking committed up to index 2
1026 14:59:08.363931 (+ 96426us) log.cc:844] Serialized 14 byte log entry
1026 14:59:08.848084 (+484153us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:08.848138 (+ 54us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:08.848181 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:08.848265 (+ 84us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:08.848379 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17172,"prepare.queue_time_us":123775,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":11125,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":10217,"thread_start_us":483714,"threads_started":1}
I20251026 14:59:08.845580 24173 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.892342 24111 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.890226 24079 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:08.857769 24114 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.907629 24170 raft_consensus.cc:2804] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:08.896585 23740 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:08.906695 24030 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:08.988098 23951 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:08.904392 22728 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:09.049392 22659 rpcz_store.cc:276] 1026 14:59:06.703415 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:07.633612 (+930197us) service_pool.cc:225] Handling call
1026 14:59:07.633923 (+ 311us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:07.634136 (+ 213us) raft_consensus.cc:1534] Early marking committed up to index 2
1026 14:59:07.661135 (+ 26999us) log.cc:844] Serialized 14 byte log entry
1026 14:59:08.895937 (+1234802us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:08.895998 (+ 61us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:08.896038 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:08.896113 (+ 75us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:08.896274 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11795,"prepare.queue_time_us":27781,"prepare.run_cpu_time_us":675,"prepare.run_wall_time_us":5114,"raft.queue_time_us":54710,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":128,"thread_start_us":1233774,"threads_started":2,"wal-append.queue_time_us":1132760}
I20251026 14:59:09.074792 24111 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:09.077932 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:09.077171 22801 raft_consensus.cc:2468] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 5.
W20251026 14:59:09.065677 22658 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.005s sys 0.019s
I20251026 14:59:08.868162 24169 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:09.085438 22730 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:09.092074 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1061) took 1977 ms. Trace:
W20251026 14:59:09.066632 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 331017us
I20251026 14:59:09.001426 23699 raft_consensus.cc:3060] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:09.078733 22732 raft_consensus.cc:2468] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
W20251026 14:59:09.079545 22658 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s user 0.006s sys 0.019s
I20251026 14:59:09.002501 24125 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:08.991066 23696 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:09.070463 23951 raft_consensus.cc:3060] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:08.894439 24087 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:08.986022 24188 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:09.006702 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 403010us
I20251026 14:59:09.086342 24085 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:09.114385 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:09.102049 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 325884us
I20251026 14:59:09.092747 22652 rpcz_store.cc:276] 1026 14:59:07.114931 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:08.104226 (+989295us) service_pool.cc:225] Handling call
1026 14:59:08.215078 (+110852us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:08.215346 (+ 268us) raft_consensus.cc:1534] Early marking committed up to index 2
1026 14:59:08.809005 (+593659us) log.cc:844] Serialized 14 byte log entry
1026 14:59:09.091374 (+282369us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:09.091496 (+ 122us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:09.091579 (+ 83us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:09.091743 (+ 164us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:09.091999 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7538,"prepare.queue_time_us":833723,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":66,"thread_start_us":800396,"threads_started":2}
I20251026 14:59:09.122689 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:09.175809 22732 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
W20251026 14:59:09.094887 24179 log.cc:927] Time spent T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.189s user 0.000s sys 0.000s
I20251026 14:59:09.056892 24142 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:09.097330 24023 raft_consensus.cc:3055] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:09.091806 24170 raft_consensus.cc:697] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:09.008705 23898 consensus_queue.cc:237] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:09.200735 22774 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:09.165226 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 66594.1us
I20251026 14:59:09.119330 24098 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:09.080703 23740 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:09.070232 24079 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:09.095530 24181 log.cc:927] Time spent T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.202s user 0.000s sys 0.000s
I20251026 14:59:09.212525 24170 consensus_queue.cc:237] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:09.213729 24183 log.cc:927] Time spent T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.143s user 0.000s sys 0.000s
I20251026 14:59:09.204682 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:09.164824 24188 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:09.215286 24188 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:09.007558 22799 rpcz_store.cc:276] 1026 14:59:07.201026 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:07.201732 (+ 706us) service_pool.cc:225] Handling call
1026 14:59:07.214770 (+ 13038us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:07.275400 (+ 60630us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:07.283886 (+ 8486us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:07.350818 (+ 66932us) log.cc:844] Serialized 26 byte log entry
1026 14:59:08.262998 (+912180us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:08.282306 (+ 19308us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:08.292037 (+ 9731us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:08.803841 (+511804us) raft_consensus.cc:1685] finished
1026 14:59:08.809920 (+ 6079us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:08.847946 (+ 38026us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9688,"raft.queue_time_us":8420,"raft.run_cpu_time_us":3214,"raft.run_wall_time_us":32820,"thread_start_us":838748,"threads_started":1,"wal-append.queue_time_us":1118556}
W20251026 14:59:09.218914 24194 log.cc:927] Time spent T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
W20251026 14:59:09.219936 24044 consensus_meta.cc:287] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.621s user 0.011s sys 0.000s
I20251026 14:59:09.210148 24209 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:09.221028 23696 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:09.221189 24209 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:09.221617 23696 consensus_queue.cc:1059] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:59:09.223073 24141 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:09.223532 24141 raft_consensus.cc:3060] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
W20251026 14:59:09.225102 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:09.210978 24191 log.cc:927] Time spent T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20251026 14:59:09.189543 24085 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:09.233992 24184 log.cc:927] Time spent T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.167s user 0.000s sys 0.001s
I20251026 14:59:09.234345 22656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1056) took 2763 ms. Trace:
W20251026 14:59:09.211741 24182 log.cc:927] Time spent T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.120s user 0.000s sys 0.000s
I20251026 14:59:09.211920 24098 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:09.181986 24103 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:09.106135 24125 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:09.201257 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:09.779531 22657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1047) took 3404 ms. Trace:
I20251026 14:59:09.854959 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 640) took 2957 ms. Trace:
W20251026 14:59:09.649852 22733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 643) took 2750 ms (client timeout 2122 ms). Trace:
I20251026 14:59:09.234719 22656 rpcz_store.cc:276] 1026 14:59:06.470744 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:07.330346 (+859602us) service_pool.cc:225] Handling call
1026 14:59:07.330610 (+ 264us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:07.495535 (+164925us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:09.234078 (+1738543us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b54001fbb54. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:09.234286 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":355454208}
I20251026 14:59:10.143041 22658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 868) took 2998 ms. Trace:
W20251026 14:59:09.942446 22659 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.837s user 0.001s sys 0.014s
W20251026 14:59:10.144340 22659 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.039s user 0.001s sys 0.018s
W20251026 14:59:09.735579 22806 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.020s sys 0.018s
I20251026 14:59:09.325632 22652 raft_consensus.cc:1240] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
W20251026 14:59:09.854265 22653 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1066) took 2467 ms (client timeout 1635 ms). Trace:
W20251026 14:59:10.013948 22774 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.809s user 0.007s sys 0.011s
W20251026 14:59:09.182277 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 385372us
W20251026 14:59:09.278885 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:10.146639 22657 rpcz_store.cc:276] 1026 14:59:06.350349 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:06.728073 (+377724us) service_pool.cc:225] Handling call
1026 14:59:07.965884 (+1237811us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:07.984307 (+ 18423us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:09.754689 (+1770382us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54001d74d0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:09.754986 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2845512064}
W20251026 14:59:10.141793 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:09.549312 22729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1043) took 3636 ms. Trace:
W20251026 14:59:10.147874 22653 rpcz_store.cc:269] 1026 14:59:07.364932 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:07.469431 (+104499us) service_pool.cc:225] Handling call
1026 14:59:09.832519 (+2363088us) spinlock_profiling.cc:243] Waited 54.4 ms on lock 0x7b540010e5d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:09.832723 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":125054848}
W20251026 14:59:10.142232 22624 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.941s (SENT) [suppressed 14 similar messages]
W20251026 14:59:10.145153 22806 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.984s user 0.022s sys 0.041s
W20251026 14:59:10.142139 22733 rpcz_store.cc:269] 1026 14:59:06.899127 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:07.478426 (+579299us) service_pool.cc:225] Handling call
1026 14:59:09.649585 (+2171159us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b540006b154. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:09.649753 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":420567296}
I20251026 14:59:10.145996 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1065) took 2841 ms. Trace:
I20251026 14:59:10.141939 22726 rpcz_store.cc:276] 1026 14:59:06.897047 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:08.298002 (+1400955us) service_pool.cc:225] Handling call
1026 14:59:08.298433 (+ 431us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:08.311287 (+ 12854us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:08.311320 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:08.311861 (+ 541us) log.cc:844] Serialized 26 byte log entry
1026 14:59:09.049414 (+737553us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:09.072964 (+ 23550us) log.cc:844] Serialized 14 byte log entry
1026 14:59:09.420092 (+347128us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:09.834377 (+414285us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b54000d66d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:09.834446 (+ 69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:09.834521 (+ 75us) raft_consensus.cc:1685] finished
1026 14:59:09.854620 (+ 20099us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:09.854867 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":47202,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":7287,"raft.queue_time_us":580300,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":225,"spinlock_wait_cycles":28519168,"thread_start_us":1050064,"threads_started":2,"wal-append.queue_time_us":968958}
W20251026 14:59:10.146900 22774 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.942s user 0.010s sys 0.011s
I20251026 14:59:10.143860 22658 rpcz_store.cc:276] 1026 14:59:07.144419 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:08.364234 (+1219815us) service_pool.cc:225] Handling call
1026 14:59:08.382779 (+ 18545us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:08.385939 (+ 3160us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:08.385977 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:08.390919 (+ 4942us) log.cc:844] Serialized 26 byte log entry
1026 14:59:09.100049 (+709130us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:09.100107 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:09.100288 (+ 181us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:10.142801 (+1042513us) raft_consensus.cc:1685] finished
1026 14:59:10.142822 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:10.142979 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2864,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":114,"thread_start_us":663887,"threads_started":1,"wal-append.queue_time_us":1058278}
I20251026 14:59:09.655596 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:46598 (request call id 0) took 2460 ms. Trace:
I20251026 14:59:09.194777 24023 raft_consensus.cc:740] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:09.187392 24199 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:09.612985 23404 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 LEARNER]: Advancing to term 3
I20251026 14:59:10.257617 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:10.271421 22806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 783) took 1508 ms. Trace:
I20251026 14:59:10.271548 22656 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:10.255620 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47996 (request call id 75) took 3290 ms. Trace:
W20251026 14:59:09.407621 22734 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s user 0.011s sys 0.017s
W20251026 14:59:09.285956 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:09.747874 22622 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.743s user 0.012s sys 0.010s
W20251026 14:59:09.739825 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:10.271818 22806 rpcz_store.cc:276] 1026 14:59:08.763443 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:08.764051 (+ 608us) service_pool.cc:225] Handling call
1026 14:59:08.764393 (+ 342us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:08.764663 (+ 270us) raft_consensus.cc:1534] Early marking committed up to index 3
1026 14:59:09.146319 (+381656us) log.cc:844] Serialized 14 byte log entry
1026 14:59:10.266711 (+1120392us) log.cc:844] Serialized 14 byte log entry
1026 14:59:10.271019 (+ 4308us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:10.271101 (+ 82us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:10.271151 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:10.271230 (+ 79us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:10.271362 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4009,"prepare.queue_time_us":1014906,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":63,"raft.queue_time_us":170,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":91,"thread_start_us":1452986,"threads_started":2}
I20251026 14:59:10.259047 22652 rpcz_store.cc:276] 1026 14:59:07.304346 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:09.190390 (+1886044us) service_pool.cc:225] Handling call
1026 14:59:09.245694 (+ 55304us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:10.145632 (+899938us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:10.145925 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:10.261116 22729 rpcz_store.cc:276] 1026 14:59:05.875141 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:06.019376 (+144235us) service_pool.cc:225] Handling call
1026 14:59:07.141369 (+1121993us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:07.254603 (+113234us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:09.402075 (+2147472us) spinlock_profiling.cc:243] Waited 65.4 ms on lock 0x7b540008d4d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:09.511533 (+109458us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b540008d4d0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:09.511729 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2687846528}
I20251026 14:59:10.262503 22651 rpcz_store.cc:276] 1026 14:59:07.200598 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:07.264143 (+ 63545us) service_pool.cc:225] Handling call
1026 14:59:09.655248 (+2391105us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b540034d354. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d11583f3 00007fd8d6109bca 00007fd8cf6b2934 00007fd8cf6b28c2 00007fd8cf6b27f1 00007fd8cf6b276c 00007fd8cf6b1a22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:09.655432 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":416624384}
I20251026 14:59:10.260501 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:10.274058 22622 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.269s user 0.012s sys 0.016s
W20251026 14:59:09.509336 24193 log.cc:927] Time spent T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1: Append to log took a long time: real 0.289s user 0.000s sys 0.000s
W20251026 14:59:09.643025 24109 consensus_meta.cc:287] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 1.293s user 0.013s sys 0.000s
I20251026 14:59:09.476073 24212 raft_consensus.cc:3055] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:09.660779 24100 raft_consensus.cc:3060] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:09.474844 22807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 770) took 2139 ms. Trace:
I20251026 14:59:09.390097 23917 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:09.655551 24200 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:10.315248 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:09.549556 24099 consensus_meta.cc:287] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 1.766s user 0.015s sys 0.013s
W20251026 14:59:10.261008 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 1.05646e+06us
I20251026 14:59:10.320547 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:10.321293 22729 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:10.323060 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:10.275774 22734 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.407s user 0.020s sys 0.045s
I20251026 14:59:10.272828 22725 rpcz_store.cc:276] 1026 14:59:06.965137 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:08.331421 (+1366284us) service_pool.cc:225] Handling call
1026 14:59:10.253501 (+1922080us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:10.253736 (+ 235us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:10.253776 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:10.253834 (+ 58us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:10.253875 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:10.253981 (+ 106us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:10.255373 (+ 1392us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b540006b150. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:10.255543 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3045010304}
I20251026 14:59:10.325737 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:09.541879 22802 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44594 (request call id 776) took 2188 ms (client timeout 1807 ms). Trace:
W20251026 14:59:09.901122 24186 log.cc:927] Time spent T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.838s user 0.001s sys 0.001s
W20251026 14:59:09.895437 24124 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.799s user 0.011s sys 0.022s
I20251026 14:59:10.270687 24032 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:09.521693 22727 consensus_meta.cc:287] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.838s user 0.010s sys 0.000s
W20251026 14:59:09.544037 24104 consensus_meta.cc:287] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.548s user 0.008s sys 0.000s
W20251026 14:59:10.112560 24166 log.cc:927] Time spent T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 1.105s user 0.003s sys 0.006s
W20251026 14:59:09.532735 24211 log.cc:927] Time spent T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.214s user 0.000s sys 0.001s
W20251026 14:59:10.279692 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:10.067198 22552 catalog_manager.cc:5649] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: term changed from 1 to 2, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4). New cstate: current_term: 2 leader_uuid: "248493ec312e4398b828c6cb00e5c4b1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20251026 14:59:09.776871 24105 log.cc:927] Time spent T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
W20251026 14:59:09.755663 22699 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.661s user 0.002s sys 0.019s
I20251026 14:59:09.856145 24050 raft_consensus.cc:1064] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e: attempting to promote NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 to VOTER
I20251026 14:59:10.287995 22732 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:09.474149 23402 raft_consensus.cc:1240] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 4 LEADER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:10.298008 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:09.690358 24115 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.839s user 0.008s sys 0.016s
W20251026 14:59:10.330641 24124 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.235s user 0.011s sys 0.035s
W20251026 14:59:10.253012 23941 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.394s user 0.005s sys 0.010s
W20251026 14:59:10.267767 23951 consensus_meta.cc:287] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 1.139s user 0.007s sys 0.003s
I20251026 14:59:09.645757 22655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1063) took 2348 ms. Trace:
I20251026 14:59:10.338184 23944 consensus_queue.cc:1059] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.119s
I20251026 14:59:10.305773 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:10.479238 22653 raft_consensus.cc:2393] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:10.479895 22653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 877) took 1060 ms. Trace:
I20251026 14:59:10.480288 22653 rpcz_store.cc:276] 1026 14:59:09.419692 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.259485 (+839793us) service_pool.cc:225] Handling call
1026 14:59:10.479817 (+220332us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:09.235195 24085 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:09.648368 24079 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:10.266933 24031 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.266s user 0.010s sys 0.036s
I20251026 14:59:09.475378 24159 consensus_queue.cc:1059] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.839s
I20251026 14:59:10.378551 23367 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters:
I20251026 14:59:09.537354 23696 raft_consensus.cc:493] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:10.263752 24062 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:10.318457 22656 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:10.511906 22656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 880) took 1091 ms. Trace:
I20251026 14:59:10.512033 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:10.512360 22656 rpcz_store.cc:276] 1026 14:59:09.420479 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.257329 (+836850us) service_pool.cc:225] Handling call
1026 14:59:10.511844 (+254515us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:10.512789 22730 raft_consensus.cc:2468] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:10.513494 22730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 665) took 1352 ms. Trace:
W20251026 14:59:10.274842 22697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.261s user 0.024s sys 0.043s
I20251026 14:59:09.476547 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:10.329514 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 638) took 3433 ms. Trace:
W20251026 14:59:10.514075 22697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.501s user 0.024s sys 0.043s
I20251026 14:59:10.514304 22734 rpcz_store.cc:276] 1026 14:59:06.896388 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:08.279601 (+1383213us) service_pool.cc:225] Handling call
1026 14:59:08.309972 (+ 30371us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:08.310247 (+ 275us) raft_consensus.cc:1534] Early marking committed up to index 2
1026 14:59:08.868382 (+558135us) log.cc:844] Serialized 14 byte log entry
1026 14:59:10.329130 (+1460748us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:10.329205 (+ 75us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:10.329266 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:10.329348 (+ 82us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:10.329460 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":59684,"prepare.queue_time_us":115003,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":52,"raft.queue_time_us":148735,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":114,"thread_start_us":1941685,"threads_started":2}
I20251026 14:59:10.326556 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 672) took 1163 ms. Trace:
I20251026 14:59:10.516430 22725 rpcz_store.cc:276] 1026 14:59:09.163605 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.325457 (+1161852us) service_pool.cc:225] Handling call
1026 14:59:10.326481 (+ 1024us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:10.517719 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:10.518446 22725 raft_consensus.cc:2468] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:59:10.519021 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 673) took 1355 ms. Trace:
I20251026 14:59:10.519397 22725 rpcz_store.cc:276] 1026 14:59:09.163905 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.517397 (+1353492us) service_pool.cc:225] Handling call
1026 14:59:10.518956 (+ 1559us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:10.520396 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:10.521924 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1064) took 3223 ms. Trace:
I20251026 14:59:10.522435 22659 rpcz_store.cc:276] 1026 14:59:07.298134 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:09.084349 (+1786215us) service_pool.cc:225] Handling call
1026 14:59:09.092792 (+ 8443us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:09.093057 (+ 265us) raft_consensus.cc:1534] Early marking committed up to index 3
1026 14:59:09.104668 (+ 11611us) log.cc:844] Serialized 14 byte log entry
1026 14:59:10.521485 (+1416817us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:10.521558 (+ 73us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:10.521618 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:10.521701 (+ 83us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:10.521852 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":363025,"prepare.queue_time_us":71655,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":9620,"thread_start_us":1039879,"threads_started":1}
I20251026 14:59:10.513921 22730 rpcz_store.cc:276] 1026 14:59:09.161500 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.511745 (+1350245us) service_pool.cc:225] Handling call
1026 14:59:10.513431 (+ 1686us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:10.393220 22699 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.309s user 0.005s sys 0.022s
I20251026 14:59:10.338145 23740 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:10.338662 24030 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:10.287503 24199 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:10.326660 23917 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:10.257524 24103 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:10.438946 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:10.440800 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:10.575409 22805 raft_consensus.cc:2393] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:10.338771 22729 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
W20251026 14:59:10.576634 22729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 675) took 1402 ms (client timeout 1678 ms). Trace:
I20251026 14:59:10.478645 23951 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:10.439604 22650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 863) took 3647 ms. Trace:
W20251026 14:59:10.331064 22802 rpcz_store.cc:269] 1026 14:59:07.353806 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:07.364655 (+ 10849us) service_pool.cc:225] Handling call
1026 14:59:09.541544 (+2176889us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b540020b2d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:09.541718 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":363323776}
W20251026 14:59:10.446468 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:10.448110 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 876) took 1028 ms. Trace:
I20251026 14:59:10.298305 24212 raft_consensus.cc:740] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:10.424060 24169 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:10.394061 24084 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:10.376843 24225 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:10.298858 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:10.624666 22657 raft_consensus.cc:2393] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:10.625099 22658 raft_consensus.cc:3060] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:10.322934 24109 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:10.467221 22731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1045) took 4581 ms. Trace:
I20251026 14:59:10.632754 22731 rpcz_store.cc:276] 1026 14:59:05.875803 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:06.035568 (+159765us) service_pool.cc:225] Handling call
1026 14:59:06.035950 (+ 382us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:06.036273 (+ 323us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:06.036299 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:06.036725 (+ 426us) log.cc:844] Serialized 26 byte log entry
1026 14:59:06.190667 (+153942us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:06.220429 (+ 29762us) log.cc:844] Serialized 14 byte log entry
1026 14:59:06.222552 (+ 2123us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:06.222650 (+ 98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:07.155521 (+932871us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b48001ec258. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d606b971 00007fd8d6062a37 00007fd8d60628ca 00007fd8d6159a0f 00007fd8d61596d7 00007fd8d616dcfc 00007fd8d616dca8 00007fd8d616dc28 00007fd8d616dbe3 00007fd8d616ce8c 00007fd8d50aa326
1026 14:59:07.509915 (+354394us) spinlock_profiling.cc:243] Waited 3.79 ms on lock 0x7b1800359860. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce07f986 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce0b1992 00007fd8ce0b3d61
1026 14:59:09.634312 (+2124397us) spinlock_profiling.cc:243] Waited 2.84 ms on lock 0x7b1800359860. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce0b1992 00007fd8ce0b3d61 00007fd8ce0b3d19 00007fd8ce0b3ca9 00007fd8ce0b3c71 00007fd8ce0b2f6d 0000000000379254 0000000000379099 00007fd8ce094d1a
1026 14:59:10.456666 (+822354us) raft_consensus.cc:1685] finished
1026 14:59:10.456701 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:10.456898 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4197,"prepare.queue_time_us":29448,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":5531,"raft.queue_time_us":16739,"raft.run_cpu_time_us":6543,"raft.run_wall_time_us":1116128,"spinlock_wait_cycles":45794304,"thread_start_us":122674,"threads_started":1,"wal-append.queue_time_us":186977}
I20251026 14:59:10.636515 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:10.637405 22731 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:10.638679 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:10.640353 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:10.314788 22807 rpcz_store.cc:276] 1026 14:59:07.279308 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:07.295638 (+ 16330us) service_pool.cc:225] Handling call
1026 14:59:07.322073 (+ 26435us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:07.581827 (+259754us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:09.390199 (+1808372us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b540004bad4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:09.415936 (+ 25737us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":410433920}
I20251026 14:59:10.643016 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:10.643797 22807 raft_consensus.cc:2468] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:59:10.338833 24032 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:10.648047 24032 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:10.471135 23402 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43606 (request call id 43) took 1262 ms. Trace:
I20251026 14:59:10.648794 23402 rpcz_store.cc:276] 1026 14:59:09.285612 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:09.298058 (+ 12446us) service_pool.cc:225] Handling call
1026 14:59:09.415719 (+117661us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:10.458048 (+1042329us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:10.471064 (+ 13016us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:10.458312 24148 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:10.342753 22727 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
W20251026 14:59:10.650516 22727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 1085) took 2114 ms (client timeout 1955 ms). Trace:
I20251026 14:59:10.650406 24148 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:10.650936 22727 rpcz_store.cc:269] 1026 14:59:08.536294 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:08.583772 (+ 47478us) service_pool.cc:225] Handling call
1026 14:59:10.650441 (+2066669us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:10.651925 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:10.652478 24148 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:10.652558 22727 raft_consensus.cc:2468] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:59:10.645053 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:10.653620 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:10.473819 22622 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:10.654294 22807 raft_consensus.cc:2393] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:10.440157 24206 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:10.654754 24206 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:10.655319 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:10.655927 22807 raft_consensus.cc:2468] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:59:10.656682 24206 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:10.656780 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:10.423841 24012 consensus_queue.cc:1059] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
W20251026 14:59:10.462122 23699 consensus_meta.cc:287] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 1.335s user 0.012s sys 0.002s
I20251026 14:59:10.657912 23699 raft_consensus.cc:515] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:10.660583 23699 leader_election.cc:290] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:10.393666 24200 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:10.455147 24115 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.604s user 0.008s sys 0.022s
I20251026 14:59:10.264294 24042 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:10.662456 24042 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:59:10.663264 24200 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:10.468775 23941 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.612s user 0.006s sys 0.010s
I20251026 14:59:10.467240 24099 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:10.503019 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:10.481231 24203 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:59:10.490974 24031 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.490s user 0.010s sys 0.037s
I20251026 14:59:10.502848 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:10.513396 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:09.474162 24216 raft_consensus.cc:2749] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251026 14:59:10.595712 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:10.595587 24141 consensus_meta.cc:287] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 1.372s user 0.012s sys 0.002s
I20251026 14:59:10.591439 24125 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:10.572021 24030 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:10.631223 24084 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:10.516568 22697 connection.cc:232] RPC call timeout handler was delayed by -1.2616763560000663s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1622s for a response
I20251026 14:59:10.524557 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:10.678539 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 871) took 1260 ms. Trace:
I20251026 14:59:10.678959 22659 rpcz_store.cc:276] 1026 14:59:09.417996 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.524230 (+1106234us) service_pool.cc:225] Handling call
1026 14:59:10.678457 (+154227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:10.281388 24188 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:09.386528 24076 consensus_queue.cc:1059] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.990s
I20251026 14:59:10.524698 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:10.690379 22730 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:10.521073 22725 raft_consensus.cc:2468] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
W20251026 14:59:10.691488 22725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 670) took 1528 ms (client timeout 1898 ms). Trace:
I20251026 14:59:10.335358 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:10.542110 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:10.493547 24142 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:10.477721 22655 rpcz_store.cc:276] 1026 14:59:07.297832 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:08.603120 (+1305288us) service_pool.cc:225] Handling call
1026 14:59:08.603522 (+ 402us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:08.662756 (+ 59234us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:08.662805 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:08.662870 (+ 65us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:08.662909 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:09.645435 (+982526us) spinlock_profiling.cc:243] Waited 47.2 ms on lock 0x7b54001fb154. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:09.645492 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:09.645681 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":108652928}
I20251026 14:59:09.404337 24205 consensus_queue.cc:1048] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.942s
I20251026 14:59:10.393332 24023 consensus_queue.cc:260] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:10.456980 24104 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:10.577123 22729 rpcz_store.cc:269] 1026 14:59:09.174556 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.284965 (+1110409us) service_pool.cc:225] Handling call
1026 14:59:10.576561 (+291596us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:10.670473 24226 log.cc:927] Time spent T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.118s user 0.000s sys 0.001s
I20251026 14:59:10.511829 24126 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:10.575062 22652 raft_consensus.cc:2468] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:59:10.320429 22728 raft_consensus.cc:2393] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:10.581928 22650 rpcz_store.cc:276] 1026 14:59:06.791811 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:07.668260 (+876449us) service_pool.cc:225] Handling call
1026 14:59:07.680542 (+ 12282us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:07.685935 (+ 5393us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:07.685979 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:07.703951 (+ 17972us) log.cc:844] Serialized 26 byte log entry
1026 14:59:08.752576 (+1048625us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:08.752638 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:08.752740 (+ 102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:10.439336 (+1686596us) raft_consensus.cc:1685] finished
1026 14:59:10.439361 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:10.439535 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":120,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":6830,"thread_start_us":994855,"threads_started":1,"wal-append.queue_time_us":1280557}
W20251026 14:59:10.516196 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 137741us
I20251026 14:59:10.585063 22651 rpcz_store.cc:276] 1026 14:59:09.419403 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.278672 (+859269us) service_pool.cc:225] Handling call
1026 14:59:10.448016 (+169344us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:10.572726 24060 raft_consensus.cc:2804] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:10.596446 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:10.804008 22805 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:10.601197 22773 connection.cc:232] RPC call timeout handler was delayed by -1.0613611040000706s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1955s for a response
W20251026 14:59:10.602593 22800 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.276s user 0.038s sys 0.081s
W20251026 14:59:10.805047 22800 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.478s user 0.038s sys 0.081s
I20251026 14:59:10.573446 24222 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:10.516043 23696 raft_consensus.cc:515] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:10.641148 24109 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:10.317158 24209 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:10.443575 24114 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:10.680162 22697 connection.cc:232] RPC call timeout handler was delayed by -1.2098820899999509s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17200000000000001s for a response
W20251026 14:59:10.873643 22697 connection.cc:232] RPC call timeout handler was delayed by -1.1893247259999953s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1582s for a response
W20251026 14:59:10.874169 22697 connection.cc:232] RPC call timeout handler was delayed by -1.163008743999967s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1746s for a response
I20251026 14:59:10.667076 22733 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:10.673064 24212 consensus_queue.cc:260] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:10.669003 22656 raft_consensus.cc:2410] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20251026 14:59:10.932642 22732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 676) took 1757 ms (client timeout 1625 ms). Trace:
W20251026 14:59:10.674152 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 200396us
I20251026 14:59:10.450719 24231 consensus_queue.cc:1059] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20251026 14:59:10.593575 23944 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:10.708382 22806 raft_consensus.cc:2410] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:10.709642 22734 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
W20251026 14:59:10.708714 22725 rpcz_store.cc:269] 1026 14:59:09.163005 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.520092 (+1357087us) service_pool.cc:225] Handling call
1026 14:59:10.691405 (+171313us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:10.938341 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:10.710650 22730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 674) took 1536 ms. Trace:
I20251026 14:59:10.939117 22730 rpcz_store.cc:276] 1026 14:59:09.174278 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.524361 (+1350083us) service_pool.cc:225] Handling call
1026 14:59:10.710562 (+186201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:10.593724 23403 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43606 (request call id 44) took 1189 ms. Trace:
I20251026 14:59:10.712194 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:10.713791 22731 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:10.715665 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:10.937348 22700 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.035s sys 0.083s
I20251026 14:59:10.440874 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:10.941118 22659 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:59:10.738938 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 873) took 1320 ms. Trace:
I20251026 14:59:10.666530 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:10.673508 24108 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:10.674310 24217 raft_consensus.cc:2804] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:10.672355 24098 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:10.669854 24206 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:10.789860 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:10.785151 23404 consensus_meta.cc:287] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 248493ec312e4398b828c6cb00e5c4b1: Time spent flushing consensus metadata: real 0.506s user 0.010s sys 0.001s
I20251026 14:59:10.672266 23951 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:10.673259 24087 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:10.676138 24030 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:10.954597 22804 raft_consensus.cc:1275] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251026 14:59:10.673250 24169 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:10.958459 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:10.958523 24169 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:10.959008 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:10.959249 22799 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:59:10.959569 22726 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:10.960160 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:10.673651 24215 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:10.665807 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:10.808835 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:10.810847 22773 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:10.838652 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:10.785204 24023 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:10.670668 24141 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:10.666747 24099 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:10.881544 24233 log.cc:927] Time spent T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
I20251026 14:59:10.881678 23740 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:10.882411 23696 leader_election.cc:290] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:10.883078 24104 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:10.883436 24224 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:10.667366 24203 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:10.885780 24230 log.cc:927] Time spent T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.260s user 0.000s sys 0.000s
W20251026 14:59:10.898170 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:10.868269 24060 raft_consensus.cc:697] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 LEADER]: Becoming Leader. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:10.785638 24142 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:10.785742 24125 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:10.923151 22803 raft_consensus.cc:1240] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:10.809064 24188 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:10.673388 24225 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:11.043236 24235 log.cc:927] Time spent T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.155s user 0.000s sys 0.000s
I20251026 14:59:11.043017 24060 consensus_queue.cc:237] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:10.928251 22805 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:11.045166 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:11.044942 24142 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:10.924269 24044 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.654s user 0.015s sys 0.014s
W20251026 14:59:11.046012 24044 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.775s user 0.015s sys 0.014s
I20251026 14:59:11.046581 24062 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:11.046977 24125 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:11.043071 24188 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:11.049774 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:11.050097 24188 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:11.050668 22803 raft_consensus.cc:2410] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:10.952416 24108 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:11.051627 23401 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:10.800889 24126 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:11.053709 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:10.939157 22808 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:10.941758 22801 raft_consensus.cc:2468] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:59:10.665263 24042 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:11.057725 24243 log.cc:927] Time spent T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.094s user 0.000s sys 0.001s
I20251026 14:59:10.942734 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:10.812492 24222 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:10.953824 23404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40908 (request call id 100) took 1403 ms. Trace:
I20251026 14:59:10.957479 24206 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:10.958014 22800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 782) took 2201 ms. Trace:
I20251026 14:59:10.940932 24212 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:10.949054 24063 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:10.952723 24217 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:11.032775 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 221960us
I20251026 14:59:11.034389 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:10.955691 24199 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:11.039878 23951 raft_consensus.cc:2804] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:10.945307 23944 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:11.047081 24062 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:11.037649 24224 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:10.931424 22733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 666) took 1769 ms (client timeout 1799 ms). Trace:
I20251026 14:59:11.053923 22805 raft_consensus.cc:2468] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
W20251026 14:59:10.933108 22732 rpcz_store.cc:269] 1026 14:59:09.174836 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:09.176712 (+ 1876us) service_pool.cc:225] Handling call
1026 14:59:10.932455 (+1755743us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54000b9fd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:10.932573 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2468827008}
W20251026 14:59:10.936043 22774 connection.cc:232] RPC call timeout handler was delayed by -2.0852590389999932s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1635s for a response
I20251026 14:59:11.059681 22806 raft_consensus.cc:1275] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:11.276741 24109 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:11.326738 24084 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:11.325117 22800 rpcz_store.cc:276] 1026 14:59:08.756778 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:08.761865 (+ 5087us) service_pool.cc:225] Handling call
1026 14:59:08.762197 (+ 332us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:08.785579 (+ 23382us) raft_consensus.cc:1534] Early marking committed up to index 2
1026 14:59:09.282899 (+497320us) log.cc:844] Serialized 14 byte log entry
1026 14:59:10.807650 (+1524751us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:10.808338 (+ 688us) log.cc:844] Serialized 26 byte log entry
1026 14:59:10.808508 (+ 170us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:10.808554 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:10.808650 (+ 96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:10.957823 (+149173us) raft_consensus.cc:1685] finished
1026 14:59:10.957843 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:10.957959 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":76553,"prepare.queue_time_us":1012607,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":37196,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":6078,"spinlock_wait_cycles":7108864,"thread_start_us":1913319,"threads_started":2,"wal-append.queue_time_us":1523373}
I20251026 14:59:11.335372 24222 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:11.339386 22804 raft_consensus.cc:2410] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:11.340008 23404 rpcz_store.cc:276] 1026 14:59:09.550402 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:09.561518 (+ 11116us) service_pool.cc:225] Handling call
1026 14:59:09.562052 (+ 534us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:10.953363 (+1391311us) raft_consensus.cc:1534] Early marking committed up to index 1
1026 14:59:10.953410 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:10.953474 (+ 64us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:10.953518 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:10.953600 (+ 82us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:10.953751 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3506,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":116}
I20251026 14:59:10.952286 24148 raft_consensus.cc:3055] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Stepping down as leader of term 2
W20251026 14:59:11.029568 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 50007.5us
I20251026 14:59:10.960950 24169 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:11.038450 24103 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:11.314288 24239 log.cc:927] Time spent T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.275s user 0.000s sys 0.000s
I20251026 14:59:10.961690 24228 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:11.008673 23917 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:11.053702 24141 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:10.981514 24171 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:11.315830 24108 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:11.313352 24225 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:10.964347 24087 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:11.361869 24125 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:59:10.981079 24215 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:11.362340 24087 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:11.134593 24099 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:11.346485 24217 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:10.955578 24153 consensus_queue.cc:1048] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.401s
I20251026 14:59:11.349323 24148 raft_consensus.cc:740] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:11.361426 24142 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:11.361810 24100 consensus_meta.cc:287] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 1.047s user 0.011s sys 0.000s
I20251026 14:59:11.364223 24215 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:11.368119 24224 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:11.366489 24227 log.cc:927] Time spent T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.908s user 0.023s sys 0.023s
I20251026 14:59:11.366429 24203 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:11.357324 24228 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:10.923880 22624 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.658s user 0.012s sys 0.020s
I20251026 14:59:10.883888 22658 raft_consensus.cc:2468] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 5.
W20251026 14:59:11.395413 22624 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.130s user 0.012s sys 0.020s
W20251026 14:59:11.396301 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 1.25392e+06us
W20251026 14:59:11.396338 22658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32888 (request call id 46) took 2131 ms (client timeout 1871 ms). Trace:
I20251026 14:59:10.950762 22654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1041) took 4609 ms. Trace:
W20251026 14:59:11.397042 22658 rpcz_store.cc:269] 1026 14:59:09.264492 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.259924 (+995432us) service_pool.cc:225] Handling call
1026 14:59:11.396004 (+1136080us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:10.950882 22622 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:11.399744 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:11.397707 22654 rpcz_store.cc:276] 1026 14:59:06.345663 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:06.345796 (+ 133us) service_pool.cc:225] Handling call
1026 14:59:07.530805 (+1185009us) spinlock_profiling.cc:243] Waited 2.2 ms on lock 0x7b5800000998. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8ce074451 00007fd8ce074302 00007fd8d6f0dad9 00007fd8d6090d30 00007fd8d615984b 00007fd8d61598d1 00007fd8d611b580 00007fd8d6105ee9 00007fd8d6105992 00007fd8cf69fd34
1026 14:59:07.542100 (+ 11295us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:07.545922 (+ 3822us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:07.545957 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:07.547175 (+ 1218us) log.cc:844] Serialized 52 byte log entry
1026 14:59:08.383369 (+836194us) raft_consensus.cc:1645] Marking committed up to 6
1026 14:59:08.879483 (+496114us) log.cc:844] Serialized 14 byte log entry
1026 14:59:09.659104 (+779621us) spinlock_profiling.cc:243] Waited 6.12 ms on lock 0x7b18002de080. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce0b1992 00007fd8ce0b3d61 00007fd8ce0b3d19 00007fd8ce0b3ca9 00007fd8ce0b3c71 00007fd8ce0b2f6d 0000000000379254 0000000000379099 00007fd8ce094d1a
1026 14:59:10.635132 (+976028us) log.cc:844] Serialized 14 byte log entry
1026 14:59:10.946564 (+311432us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:10.946674 (+ 110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:10.946735 (+ 61us) raft_consensus.cc:1685] finished
1026 14:59:10.946754 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:10.946920 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":716841,"prepare.queue_time_us":938766,"prepare.run_cpu_time_us":695,"prepare.run_wall_time_us":12864,"raft.queue_time_us":224141,"raft.run_cpu_time_us":440,"raft.run_wall_time_us":8272,"spinlock_wait_cycles":19128448,"thread_start_us":2378127,"threads_started":5,"wal-append.queue_time_us":1449911,"wal-append.run_cpu_time_us":4173,"wal-append.run_wall_time_us":1348390}
I20251026 14:59:10.933053 22656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 882) took 1512 ms. Trace:
I20251026 14:59:11.401762 22656 rpcz_store.cc:276] 1026 14:59:09.421058 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.513087 (+1092029us) service_pool.cc:225] Handling call
1026 14:59:10.932995 (+419908us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.402843 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:11.403262 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:59:10.955590 22657 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 885) took 1472 ms (client timeout 1653 ms). Trace:
I20251026 14:59:11.404389 22656 raft_consensus.cc:2376] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3. Re-sending same reply.
I20251026 14:59:10.941803 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 878) took 1521 ms. Trace:
I20251026 14:59:10.961625 22651 raft_consensus.cc:2468] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:10.876381 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:11.404796 22657 rpcz_store.cc:269] 1026 14:59:09.483060 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.260216 (+777156us) service_pool.cc:225] Handling call
1026 14:59:10.955412 (+695196us) spinlock_profiling.cc:243] Waited 319 ms on lock 0x7b540012e8d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:10.955533 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":733121408}
I20251026 14:59:11.407681 22650 raft_consensus.cc:2393] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:11.407845 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:11.408231 22650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 888) took 1015 ms. Trace:
I20251026 14:59:10.941789 22652 rpcz_store.cc:276] 1026 14:59:09.418552 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.325294 (+906742us) service_pool.cc:225] Handling call
1026 14:59:10.738862 (+413568us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.406530 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1070) took 1130 ms. Trace:
I20251026 14:59:11.405691 22659 rpcz_store.cc:276] 1026 14:59:09.419950 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.679774 (+1259824us) service_pool.cc:225] Handling call
1026 14:59:10.941746 (+261972us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.409297 22651 rpcz_store.cc:276] 1026 14:59:10.276171 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.799133 (+522962us) service_pool.cc:225] Handling call
1026 14:59:11.406313 (+607180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.410039 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:11.409910 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:11.410640 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:11.411224 22659 raft_consensus.cc:2393] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:11.411667 22651 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:11.412155 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:11.412307 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:11.406044 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:11.412961 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1069) took 1137 ms. Trace:
I20251026 14:59:11.422212 22731 raft_consensus.cc:2410] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20251026 14:59:11.400779 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 449831us
I20251026 14:59:11.422696 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:11.423118 22731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 671) took 2259 ms (client timeout 2833 ms). Trace:
I20251026 14:59:11.423743 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 887) took 1042 ms. Trace:
I20251026 14:59:10.944643 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:10.668360 22653 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:11.425190 22725 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:11.425707 22734 raft_consensus.cc:2410] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20251026 14:59:11.425956 22653 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 884) took 1943 ms (client timeout 2092 ms). Trace:
W20251026 14:59:11.426489 22734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 667) took 2264 ms (client timeout 1822 ms). Trace:
I20251026 14:59:11.424712 22622 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:11.427116 22733 rpcz_store.cc:269] 1026 14:59:09.161805 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.502749 (+1340944us) service_pool.cc:225] Handling call
1026 14:59:10.931362 (+428613us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:11.426505 22653 rpcz_store.cc:269] 1026 14:59:09.482758 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.481115 (+998357us) service_pool.cc:225] Handling call
1026 14:59:11.425727 (+944612us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.428114 22622 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:11.428556 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:11.408607 22650 rpcz_store.cc:276] 1026 14:59:10.393121 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.804851 (+411730us) service_pool.cc:225] Handling call
1026 14:59:11.408161 (+603310us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.422283 22652 rpcz_store.cc:276] 1026 14:59:10.275850 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.411735 (+1135885us) service_pool.cc:225] Handling call
1026 14:59:11.412883 (+ 1148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.430007 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:11.415597 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:11.430119 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:11.431042 22653 raft_consensus.cc:2468] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
W20251026 14:59:10.813380 22728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 1093) took 2214 ms (client timeout 2082 ms). Trace:
W20251026 14:59:11.432253 22728 rpcz_store.cc:269] 1026 14:59:08.599117 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:09.090304 (+491187us) service_pool.cc:225] Handling call
1026 14:59:10.808935 (+1718631us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:10.945744 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:11.433490 22726 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:11.434070 22730 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:11.434207 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 682) took 1167 ms. Trace:
I20251026 14:59:11.434641 22730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 1096) took 1129 ms. Trace:
I20251026 14:59:11.420580 22729 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:11.435097 22730 rpcz_store.cc:276] 1026 14:59:10.305297 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.945458 (+640161us) service_pool.cc:225] Handling call
1026 14:59:11.434578 (+489120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.436928 22730 raft_consensus.cc:1240] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251026 14:59:11.437674 22730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1076) took 4435 ms. Trace:
W20251026 14:59:11.437848 22624 connection.cc:232] RPC call timeout handler was delayed by -1.9944012539999676s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1898s for a response
I20251026 14:59:11.438123 22730 rpcz_store.cc:276] 1026 14:59:07.002472 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.435951 (+4433479us) service_pool.cc:225] Handling call
1026 14:59:11.436508 (+ 557us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:11.437337 (+ 829us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.437568 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.438589 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:11.439772 22652 raft_consensus.cc:2393] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:11.440205 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 886) took 1059 ms. Trace:
I20251026 14:59:11.440526 22652 rpcz_store.cc:276] 1026 14:59:10.380551 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.437641 (+1057090us) service_pool.cc:225] Handling call
1026 14:59:11.440145 (+ 2504us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.420640 22727 raft_consensus.cc:2468] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:59:11.426385 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:11.442178 22732 raft_consensus.cc:2468] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
W20251026 14:59:11.414644 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 524660us
W20251026 14:59:11.426970 22734 rpcz_store.cc:269] 1026 14:59:09.162118 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.516258 (+1354140us) service_pool.cc:225] Handling call
1026 14:59:11.426359 (+910101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.442780 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 1095) took 1155 ms. Trace:
I20251026 14:59:11.443176 22732 rpcz_store.cc:276] 1026 14:59:10.286983 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.425576 (+1138593us) service_pool.cc:225] Handling call
1026 14:59:11.442709 (+ 17133us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:11.431380 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:11.445096 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 652) took 4236 ms. Trace:
W20251026 14:59:10.941449 22700 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.556s user 0.036s sys 0.083s
W20251026 14:59:11.423640 22731 rpcz_store.cc:269] 1026 14:59:09.163300 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.639992 (+1476692us) service_pool.cc:225] Handling call
1026 14:59:11.422927 (+782935us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.446521 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:11.428586 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:11.446720 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:11.447652 22733 raft_consensus.cc:2468] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:11.431692 22650 raft_consensus.cc:2410] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:11.414413 22656 raft_consensus.cc:2393] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:11.441983 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:11.412336 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1073) took 1129 ms. Trace:
W20251026 14:59:11.416981 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:11.404369 22658 raft_consensus.cc:2393] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:11.458886 22651 rpcz_store.cc:276] 1026 14:59:10.283208 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.410211 (+1127003us) service_pool.cc:225] Handling call
1026 14:59:11.412278 (+ 2067us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.434538 22726 rpcz_store.cc:276] 1026 14:59:10.267030 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.958778 (+691748us) service_pool.cc:225] Handling call
1026 14:59:11.434125 (+475347us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.460037 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:11.448233 22733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 1094) took 1164 ms. Trace:
I20251026 14:59:11.460230 22725 raft_consensus.cc:2468] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:11.460619 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:11.445818 22732 rpcz_store.cc:276] 1026 14:59:07.208386 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.443896 (+4235510us) service_pool.cc:225] Handling call
1026 14:59:11.444364 (+ 468us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.444731 (+ 367us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:11.444756 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:11.444807 (+ 51us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:11.444856 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.444907 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:11.445043 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.459033 22728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 679) took 1317 ms. Trace:
I20251026 14:59:11.461650 22728 rpcz_store.cc:276] 1026 14:59:10.141824 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.441452 (+1299628us) service_pool.cc:225] Handling call
1026 14:59:11.458969 (+ 17517us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:11.462531 22697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:11.462845 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 660) took 4126 ms. Trace:
I20251026 14:59:11.437371 22659 rpcz_store.cc:276] 1026 14:59:10.381262 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.422163 (+1040901us) service_pool.cc:225] Handling call
1026 14:59:11.423672 (+ 1509us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.463351 22732 rpcz_store.cc:276] 1026 14:59:07.336682 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.461634 (+4124952us) service_pool.cc:225] Handling call
1026 14:59:11.462077 (+ 443us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.462514 (+ 437us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:11.462542 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:11.462605 (+ 63us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:11.462633 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.462691 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:11.462792 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.463904 22728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47996 (request call id 76) took 4127 ms. Trace:
I20251026 14:59:11.460880 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 677) took 1319 ms. Trace:
I20251026 14:59:11.453425 22700 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:11.464694 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:11.466197 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1072) took 1183 ms. Trace:
I20251026 14:59:11.466432 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:11.466595 22659 rpcz_store.cc:276] 1026 14:59:10.282767 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.464319 (+1181552us) service_pool.cc:225] Handling call
1026 14:59:11.466126 (+ 1807us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.467746 22658 raft_consensus.cc:2410] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:11.468379 22700 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:11.468834 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32888 (request call id 44) took 3758 ms. Trace:
I20251026 14:59:11.448143 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:11.469732 22658 raft_consensus.cc:3055] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:11.460726 22733 rpcz_store.cc:276] 1026 14:59:10.283877 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.428201 (+1144324us) service_pool.cc:225] Handling call
1026 14:59:11.448175 (+ 19974us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.459131 22650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 889) took 1016 ms. Trace:
I20251026 14:59:11.471841 22650 rpcz_store.cc:276] 1026 14:59:10.442553 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.429738 (+987185us) service_pool.cc:225] Handling call
1026 14:59:11.459055 (+ 29317us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.472710 22700 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:11.469754 22653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 872) took 2051 ms. Trace:
I20251026 14:59:11.469329 22659 rpcz_store.cc:276] 1026 14:59:07.710484 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.467549 (+3757065us) service_pool.cc:225] Handling call
1026 14:59:11.468043 (+ 494us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.468401 (+ 358us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:11.468428 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:11.468486 (+ 58us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:11.468533 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.468621 (+ 88us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:11.468735 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.473847 22733 raft_consensus.cc:1240] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:11.474432 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 875) took 2055 ms. Trace:
I20251026 14:59:11.474839 22659 raft_consensus.cc:1240] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:11.474984 22733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1078) took 3035 ms. Trace:
I20251026 14:59:11.476135 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1068) took 1200 ms. Trace:
I20251026 14:59:11.471513 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 661) took 4106 ms. Trace:
I20251026 14:59:11.481525 22732 rpcz_store.cc:276] 1026 14:59:07.364932 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.464679 (+4099747us) service_pool.cc:225] Handling call
1026 14:59:11.470743 (+ 6064us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.471128 (+ 385us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:11.471154 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:11.471208 (+ 54us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:11.471253 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.471311 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:11.471442 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.485311 22732 raft_consensus.cc:1240] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:11.487556 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1079) took 3047 ms. Trace:
I20251026 14:59:11.488864 22732 rpcz_store.cc:276] 1026 14:59:08.440060 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.484577 (+3044517us) service_pool.cc:225] Handling call
1026 14:59:11.485068 (+ 491us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.487234 (+ 2166us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.487489 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.464941 22725 rpcz_store.cc:276] 1026 14:59:10.141206 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.944325 (+803119us) service_pool.cc:225] Handling call
1026 14:59:11.460811 (+516486us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.495144 22732 raft_consensus.cc:1240] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:11.495949 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1080) took 3055 ms. Trace:
I20251026 14:59:11.496104 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1084) took 2974 ms. Trace:
I20251026 14:59:11.476650 22658 raft_consensus.cc:740] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:11.476651 22659 rpcz_store.cc:276] 1026 14:59:10.275563 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.474245 (+1198682us) service_pool.cc:225] Handling call
1026 14:59:11.474647 (+ 402us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.475858 (+ 1211us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.476082 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.476106 22733 rpcz_store.cc:276] 1026 14:59:08.439717 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.471381 (+3031664us) service_pool.cc:225] Handling call
1026 14:59:11.473579 (+ 2198us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.474663 (+ 1084us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.474911 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:11.480885 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:11.498608 22659 raft_consensus.cc:1240] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:11.498750 22733 raft_consensus.cc:1240] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:11.482415 22773 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:11.499439 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1074) took 1171 ms. Trace:
I20251026 14:59:11.499998 22659 rpcz_store.cc:276] 1026 14:59:10.328416 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.497813 (+1169397us) service_pool.cc:225] Handling call
1026 14:59:11.498351 (+ 538us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.499132 (+ 781us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.499370 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:11.483745 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:11.499439 22733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1089) took 2907 ms. Trace:
I20251026 14:59:11.490301 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:11.490602 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:11.501796 22733 rpcz_store.cc:276] 1026 14:59:08.592462 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.498099 (+2905637us) service_pool.cc:225] Handling call
1026 14:59:11.498490 (+ 391us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.499210 (+ 720us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.499386 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.503099 22800 raft_consensus.cc:2410] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:11.503273 22799 raft_consensus.cc:2410] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:11.503428 22733 raft_consensus.cc:1240] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:11.504184 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:11.504448 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:11.504487 22733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 664) took 2343 ms. Trace:
I20251026 14:59:11.505494 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:11.496784 22725 rpcz_store.cc:276] 1026 14:59:08.521739 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.494589 (+2972850us) service_pool.cc:225] Handling call
1026 14:59:11.495124 (+ 535us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.495583 (+ 459us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:11.495631 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:11.495722 (+ 91us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:11.495778 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.495861 (+ 83us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:11.496025 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.506120 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:11.507133 22799 raft_consensus.cc:2468] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:11.507548 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:11.507622 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:11.508790 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:11.509181 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:11.509398 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:11.510253 22803 raft_consensus.cc:1240] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:11.491511 22650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 879) took 2071 ms. Trace:
I20251026 14:59:11.500391 22773 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:11.510867 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:11.112912 22807 raft_consensus.cc:2410] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:11.464766 22728 rpcz_store.cc:276] 1026 14:59:07.336894 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.462517 (+4125623us) service_pool.cc:225] Handling call
1026 14:59:11.463017 (+ 500us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.463484 (+ 467us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:11.463514 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:11.463581 (+ 67us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:11.463621 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.463716 (+ 95us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:11.463849 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.460842 22656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 881) took 2040 ms. Trace:
I20251026 14:59:11.476488 22652 raft_consensus.cc:1240] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
W20251026 14:59:11.351841 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 310066us
I20251026 14:59:11.514398 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1071) took 1237 ms. Trace:
W20251026 14:59:11.514391 23402 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.004s sys 0.029s
I20251026 14:59:11.514844 23404 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:10.940691 23403 rpcz_store.cc:276] 1026 14:59:09.419402 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:09.476206 (+ 56804us) service_pool.cc:225] Handling call
1026 14:59:10.581636 (+1105430us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.515017 22652 rpcz_store.cc:276] 1026 14:59:10.276498 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.475952 (+1199454us) service_pool.cc:225] Handling call
1026 14:59:11.476318 (+ 366us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.514041 (+ 37723us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.514297 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.512005 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:11.510630 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:11.516602 22804 raft_consensus.cc:2410] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20251026 14:59:11.515174 23402 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.537s user 0.005s sys 0.030s
I20251026 14:59:11.511965 22773 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:11.367919 24148 consensus_queue.cc:260] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:11.343538 23951 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:11.475124 22653 rpcz_store.cc:276] 1026 14:59:09.418293 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.447807 (+2029514us) service_pool.cc:225] Handling call
1026 14:59:11.469696 (+ 21889us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.516867 22806 raft_consensus.cc:2410] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20251026 14:59:11.518420 22624 connection.cc:232] RPC call timeout handler was delayed by -1.81765146500004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2016s for a response
I20251026 14:59:11.496428 22732 rpcz_store.cc:276] 1026 14:59:08.440376 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.494322 (+3053946us) service_pool.cc:225] Handling call
1026 14:59:11.494845 (+ 523us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.495648 (+ 803us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.495896 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.512229 22808 raft_consensus.cc:2468] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:59:11.516901 22650 rpcz_store.cc:276] 1026 14:59:09.420198 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.472683 (+2052485us) service_pool.cc:225] Handling call
1026 14:59:11.473121 (+ 438us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.490953 (+ 17832us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:11.490989 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:11.491069 (+ 80us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:11.491109 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.491186 (+ 77us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:11.491417 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14817}
I20251026 14:59:11.517694 22656 rpcz_store.cc:276] 1026 14:59:09.420774 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.459646 (+2038872us) service_pool.cc:225] Handling call
1026 14:59:11.460779 (+ 1133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.517340 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 890) took 1070 ms. Trace:
I20251026 14:59:11.456167 22729 raft_consensus.cc:2468] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:11.505570 22733 rpcz_store.cc:276] 1026 14:59:09.161112 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.502781 (+2341669us) service_pool.cc:225] Handling call
1026 14:59:11.503187 (+ 406us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.503896 (+ 709us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.504431 (+ 535us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.510340 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:11.346553 24063 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:11.474808 22651 rpcz_store.cc:276] 1026 14:59:09.419111 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.460273 (+2041162us) service_pool.cc:225] Handling call
1026 14:59:11.474363 (+ 14090us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.493405 22622 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:11.525080 22659 raft_consensus.cc:1275] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:11.527560 22728 raft_consensus.cc:1275] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:11.541188 24030 raft_consensus.cc:2804] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:11.542065 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 669) took 2379 ms. Trace:
I20251026 14:59:11.542373 24100 raft_consensus.cc:2804] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:11.545920 23944 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:11.546607 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:11.547228 24050 consensus_queue.cc:237] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:11.026700 23740 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:11.549633 24062 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:11.733147 22725 rpcz_store.cc:276] 1026 14:59:09.162702 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.535524 (+2372822us) service_pool.cc:225] Handling call
1026 14:59:11.541002 (+ 5478us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.541460 (+ 458us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:11.541505 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:11.541593 (+ 88us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:11.541638 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.541729 (+ 91us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:11.541946 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.675237 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:11.735097 22652 rpcz_store.cc:276] 1026 14:59:10.447823 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.515990 (+1068167us) service_pool.cc:225] Handling call
1026 14:59:11.516643 (+ 653us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.516969 (+ 326us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:11.516995 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:11.517058 (+ 63us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:11.517084 (+ 26us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.517141 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:11.517264 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:11.566895 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 55155.3us
I20251026 14:59:11.744211 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 892) took 1263 ms. Trace:
W20251026 14:59:11.742863 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 65410.6us
I20251026 14:59:11.745023 22659 rpcz_store.cc:276] 1026 14:59:10.476195 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.517112 (+1040917us) service_pool.cc:225] Handling call
1026 14:59:11.517676 (+ 564us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.734983 (+217307us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.739104 (+ 4121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.364228 24087 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:11.698431 24099 raft_consensus.cc:2804] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:11.732921 24100 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:11.749955 22655 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 883) took 2328 ms (client timeout 2200 ms). Trace:
I20251026 14:59:11.761019 24085 raft_consensus.cc:3055] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:11.761543 24085 raft_consensus.cc:740] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:11.762480 23740 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:11.762976 24115 raft_consensus.cc:3055] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:11.762674 24085 consensus_queue.cc:260] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:11.763938 24085 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:11.734509 24030 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:11.567551 24109 raft_consensus.cc:2804] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:11.573649 24104 raft_consensus.cc:2804] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:11.567605 24203 raft_consensus.cc:2804] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:11.557752 24138 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:11.749610 24125 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:11.552835 24108 raft_consensus.cc:2804] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251026 14:59:11.759734 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 287140us
I20251026 14:59:11.553541 24076 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:11.570597 24224 raft_consensus.cc:2804] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:11.569929 24222 raft_consensus.cc:2804] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20251026 14:59:11.765355 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 272265us
I20251026 14:59:11.558568 24073 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:11.764928 24100 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:11.769019 24237 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:11.560372 24199 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:11.770156 22624 connection.cc:232] RPC call timeout handler was delayed by -1.6734589419999111s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2209s for a response
I20251026 14:59:11.797712 24225 consensus_queue.cc:1048] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:11.798123 22700 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:11.760993 22655 rpcz_store.cc:269] 1026 14:59:09.482317 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.719405 (+1237088us) service_pool.cc:225] Handling call
1026 14:59:11.546280 (+826875us) spinlock_profiling.cc:243] Waited 4.07 ms on lock 0x7b5800000998. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8ce074451 00007fd8ce0747ce 000000000035a3c9 00007fd8d6091030 00007fd8d6159864 00007fd8d61598d1 00007fd8d6107660 00007fd8d6106f44 00007fd8cf6a4834 00007fd8cf6a47c2
1026 14:59:11.749868 (+203588us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9371392}
I20251026 14:59:11.793473 24108 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:11.806082 22799 raft_consensus.cc:2410] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:11.805971 22730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 648) took 4602 ms. Trace:
I20251026 14:59:11.803022 24138 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:11.806561 24169 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:11.763427 24115 raft_consensus.cc:740] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:11.758796 22728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 668) took 2584 ms. Trace:
I20251026 14:59:11.806612 22658 consensus_queue.cc:260] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:11.808830 22654 raft_consensus.cc:2368] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4.
I20251026 14:59:11.758291 24099 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:11.768951 24109 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:11.828246 24109 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:11.768141 24030 raft_consensus.cc:3060] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:11.752820 24217 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:11.772221 22802 consensus_meta.cc:287] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.816s user 0.018s sys 0.000s
I20251026 14:59:11.814452 24073 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:11.830027 24199 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:11.815157 24258 consensus_queue.cc:1048] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251026 14:59:11.807581 24222 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
W20251026 14:59:11.816300 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 46113us
W20251026 14:59:11.816545 22727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 1100) took 1437 ms (client timeout 1688 ms). Trace:
I20251026 14:59:11.816013 24237 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:11.845770 22727 rpcz_store.cc:269] 1026 14:59:10.379727 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.653367 (+273640us) service_pool.cc:225] Handling call
1026 14:59:11.816253 (+1162886us) spinlock_profiling.cc:243] Waited 766 ms on lock 0x7b54000b90d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:11.816466 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1761573760}
I20251026 14:59:11.846657 24076 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:11.847524 24237 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:11.850687 22725 raft_consensus.cc:1240] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:11.850476 24030 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:11.851429 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1099) took 1521 ms. Trace:
I20251026 14:59:11.851815 22725 rpcz_store.cc:276] 1026 14:59:10.329624 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.850122 (+1520498us) service_pool.cc:225] Handling call
1026 14:59:11.850460 (+ 338us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.851170 (+ 710us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.851375 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.853322 24030 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:11.771421 24104 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:11.856848 24104 raft_consensus.cc:3060] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:11.817970 24062 consensus_queue.cc:1059] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:11.823173 22732 raft_consensus.cc:1240] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:11.807782 24224 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
W20251026 14:59:11.824457 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:11.827564 24126 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:11.827559 24099 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
W20251026 14:59:11.830427 22624 connection.cc:232] RPC call timeout handler was delayed by -1.4859623019999617s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1517s for a response
I20251026 14:59:11.831229 22658 raft_consensus.cc:3060] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:11.831764 22654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1080) took 1358 ms. Trace:
I20251026 14:59:11.833397 22622 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:11.837965 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 654) took 4629 ms. Trace:
I20251026 14:59:11.862713 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1098) took 1534 ms. Trace:
I20251026 14:59:11.822165 24203 raft_consensus.cc:697] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:11.822377 24108 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:11.834216 22733 raft_consensus.cc:1275] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:59:11.864203 22733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 683) took 1588 ms. Trace:
I20251026 14:59:11.864707 22733 rpcz_store.cc:276] 1026 14:59:10.276310 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.742206 (+1465896us) service_pool.cc:225] Handling call
1026 14:59:11.810040 (+ 67834us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.863833 (+ 53793us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.864132 (+ 299us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.822218 22730 rpcz_store.cc:276] 1026 14:59:07.203316 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.439588 (+4236272us) service_pool.cc:225] Handling call
1026 14:59:11.440019 (+ 431us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:11.440569 (+ 550us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:11.440592 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:11.459632 (+ 19040us) log.cc:844] Serialized 26 byte log entry
1026 14:59:11.536229 (+ 76597us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:11.536298 (+ 69us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.536396 (+ 98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:11.805626 (+269230us) raft_consensus.cc:1685] finished
1026 14:59:11.805660 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:11.805885 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21206,"raft.queue_time_us":92421,"raft.run_cpu_time_us":846,"raft.run_wall_time_us":18955,"spinlock_wait_cycles":883584,"thread_start_us":76144,"threads_started":1,"wal-append.queue_time_us":309710}
I20251026 14:59:11.844311 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 649) took 4637 ms. Trace:
I20251026 14:59:11.772640 24032 raft_consensus.cc:2804] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251026 14:59:11.859534 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:11.845584 24023 consensus_meta.cc:287] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.817s user 0.013s sys 0.001s
W20251026 14:59:11.862180 22624 connection.cc:232] RPC call timeout handler was delayed by -1.1529012539999712s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.28330000000000005s for a response
I20251026 14:59:11.862428 22654 rpcz_store.cc:276] 1026 14:59:10.472902 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.674716 (+1201814us) service_pool.cc:225] Handling call
1026 14:59:11.831668 (+156952us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.839473 23951 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:11.862602 22726 rpcz_store.cc:276] 1026 14:59:07.209016 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.461472 (+4252456us) service_pool.cc:225] Handling call
1026 14:59:11.461915 (+ 443us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:11.462454 (+ 539us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:11.462498 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:11.463217 (+ 719us) log.cc:844] Serialized 52 byte log entry
1026 14:59:11.536580 (+ 73363us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:11.536626 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.536703 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:11.837729 (+301026us) raft_consensus.cc:1685] finished
1026 14:59:11.837751 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:11.837910 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":101713,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":227,"thread_start_us":72905,"threads_started":1,"wal-append.queue_time_us":347384}
I20251026 14:59:11.845255 24222 raft_consensus.cc:3060] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:11.871755 22657 raft_consensus.cc:2393] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:11.863477 22732 rpcz_store.cc:276] 1026 14:59:10.324806 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.745404 (+1420598us) service_pool.cc:225] Handling call
1026 14:59:11.809601 (+ 64197us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.858741 (+ 49140us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.859011 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.826840 22728 rpcz_store.cc:276] 1026 14:59:09.162406 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.526363 (+2363957us) service_pool.cc:225] Handling call
1026 14:59:11.526888 (+ 525us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.735547 (+208659us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.747222 (+ 11675us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.861434 22802 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
W20251026 14:59:11.885833 22802 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 881) took 1428 ms (client timeout 1622 ms). Trace:
W20251026 14:59:11.886245 22802 rpcz_store.cc:269] 1026 14:59:10.457692 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.631229 (+173537us) service_pool.cc:225] Handling call
1026 14:59:11.885726 (+1254497us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:11.886041 24203 consensus_queue.cc:237] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:11.866729 22734 rpcz_store.cc:276] 1026 14:59:07.207385 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.443322 (+4235937us) service_pool.cc:225] Handling call
1026 14:59:11.443797 (+ 475us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:11.444447 (+ 650us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:11.444504 (+ 57us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:11.445254 (+ 750us) log.cc:844] Serialized 52 byte log entry
1026 14:59:11.535684 (+ 90430us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:11.843714 (+308030us) log.cc:844] Serialized 14 byte log entry
1026 14:59:11.843977 (+ 263us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.844051 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:11.844096 (+ 45us) raft_consensus.cc:1685] finished
1026 14:59:11.844107 (+ 11us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:11.844244 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9309,"raft.queue_time_us":94519,"raft.run_cpu_time_us":411,"raft.run_wall_time_us":176057,"thread_start_us":183530,"threads_started":2,"wal-append.queue_time_us":363607}
W20251026 14:59:11.889488 22624 connection.cc:232] RPC call timeout handler was delayed by -1.0494629390000227s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.193s for a response
W20251026 14:59:11.890228 22657 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 870) took 2472 ms (client timeout 2713 ms). Trace:
W20251026 14:59:11.890681 22657 rpcz_store.cc:269] 1026 14:59:09.417678 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.407430 (+1989752us) service_pool.cc:225] Handling call
1026 14:59:11.890008 (+482578us) spinlock_profiling.cc:243] Waited 455 ms on lock 0x7b54001581d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:11.890148 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1047119104}
I20251026 14:59:11.823439 24125 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:11.865470 24076 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:11.890453 24217 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:11.877832 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:11.878576 22800 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:11.892892 24217 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:11.860126 24224 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:11.893651 24217 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:59:11.895874 24217 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:11.837064 24115 consensus_queue.cc:260] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:11.869256 24032 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:11.897450 24115 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:11.882948 22700 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:11.899490 24237 raft_consensus.cc:2749] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:11.891119 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:11.897491 24032 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:11.870280 24225 raft_consensus.cc:2804] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:11.903890 24214 raft_consensus.cc:3055] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:11.904623 24214 raft_consensus.cc:740] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:11.903723 22655 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 5.
I20251026 14:59:11.905682 24214 consensus_queue.cc:260] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:11.906778 24214 raft_consensus.cc:3060] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:11.904228 24225 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:11.907260 24225 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:11.908634 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:11.903774 24199 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:11.911324 24219 log.cc:927] Time spent T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.985s user 0.004s sys 0.004s
W20251026 14:59:11.912011 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 102108us
I20251026 14:59:11.914088 24031 raft_consensus.cc:2804] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:11.914574 24031 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:11.914907 24031 raft_consensus.cc:3060] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:11.917167 22700 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:11.917264 24030 consensus_queue.cc:1059] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:11.918233 24030 consensus_queue.cc:1243] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer 0d684445287b450a8f738b6ad4b59821 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251026 14:59:11.919513 24103 raft_consensus.cc:3055] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:11.919903 24103 raft_consensus.cc:740] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:11.920526 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:11.918159 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:11.885426 22773 connection.cc:232] RPC call timeout handler was delayed by -1.1576039550000132s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1688s for a response
W20251026 14:59:11.922112 22773 connection.cc:232] RPC call timeout handler was delayed by -1.1504039550000016s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1696s for a response
I20251026 14:59:11.927313 24138 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:11.924453 24103 consensus_queue.cc:260] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:11.863453 24169 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:11.928493 24103 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:11.928936 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:11.929785 24169 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:11.930354 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:11.931674 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:11.932217 24079 raft_consensus.cc:3055] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:11.932696 24079 raft_consensus.cc:740] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:11.896972 24063 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:11.932355 22806 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:11.930498 22805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 896) took 1055 ms. Trace:
I20251026 14:59:11.933691 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40908 (request call id 99) took 2383 ms. Trace:
I20251026 14:59:11.937170 22805 rpcz_store.cc:276] 1026 14:59:10.875159 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.044913 (+169754us) service_pool.cc:225] Handling call
1026 14:59:11.930178 (+885265us) spinlock_profiling.cc:243] Waited 8.05 ms on lock 0x7b540004c9d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:11.930340 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18518912}
W20251026 14:59:11.944623 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:11.948015 22658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32888 (request call id 47) took 2553 ms. Trace:
I20251026 14:59:11.948357 22658 rpcz_store.cc:276] 1026 14:59:09.389633 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.468550 (+2078917us) service_pool.cc:225] Handling call
1026 14:59:11.469510 (+ 960us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.942312 (+472802us) raft_consensus.cc:1534] Early marking committed up to index 5
1026 14:59:11.942340 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:11.942368 (+ 28us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:11.942386 (+ 18us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.942431 (+ 45us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:11.942515 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":15543}
I20251026 14:59:11.941265 24258 consensus_queue.cc:1059] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
W20251026 14:59:11.950208 22773 connection.cc:232] RPC call timeout handler was delayed by -1.0001039549999859s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18630000000000002s for a response
W20251026 14:59:11.950829 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:11.951354 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:11.937963 23400 rpcz_store.cc:276] 1026 14:59:09.549994 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:09.550201 (+ 207us) service_pool.cc:225] Handling call
1026 14:59:09.550581 (+ 380us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:10.631146 (+1080565us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:10.665931 (+ 34785us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:10.665976 (+ 45us) raft_consensus.cc:1645] Marking committed up to 6
1026 14:59:10.666031 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.933141 (+1267110us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540056e0d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:11.933251 (+ 110us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:11.933481 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2400822656}
W20251026 14:59:11.953483 22729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 680) took 1811 ms (client timeout 1930 ms). Trace:
W20251026 14:59:11.953902 22729 rpcz_store.cc:269] 1026 14:59:10.142098 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:10.715313 (+573215us) service_pool.cc:225] Handling call
1026 14:59:11.953179 (+1237866us) spinlock_profiling.cc:243] Waited 479 ms on lock 0x7b54000ba4d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:11.953414 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1100846848}
I20251026 14:59:11.954281 24258 raft_consensus.cc:3055] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:11.954692 24258 raft_consensus.cc:740] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:11.955641 24258 consensus_queue.cc:260] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:11.956701 24073 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:11.956907 24258 raft_consensus.cc:3060] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Advancing to term 3
W20251026 14:59:11.957638 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:11.975056 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 57959.8us
W20251026 14:59:11.977459 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 182704us
W20251026 14:59:11.978658 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:11.969549 24169 raft_consensus.cc:2749] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:11.985208 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:11.989183 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:11.989861 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:11.990305 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:11.990404 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:11.991451 22802 raft_consensus.cc:1240] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:11.993755 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:11.995222 24079 consensus_queue.cc:260] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:12.007370 22774 leader_election.cc:336] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Timed out: RequestConsensusVote RPC to 127.21.2.1:46185 timed out after 1.612s (SENT)
I20251026 14:59:12.007992 22774 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:11.978518 24109 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.011071 24109 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:11.984822 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.012004 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.012629 22803 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:12.013049 22799 raft_consensus.cc:2393] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:12.015240 22808 raft_consensus.cc:2368] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5.
I20251026 14:59:11.993006 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:12.015628 22731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 653) took 4806 ms. Trace:
I20251026 14:59:12.017292 24073 raft_consensus.cc:2749] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:12.025947 24206 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:12.041128 24125 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:12.049659 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:12.031273 24114 raft_consensus.cc:3055] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:12.043334 24104 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.037658 23400 raft_consensus.cc:1275] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 LEARNER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251026 14:59:12.032583 24173 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:12.039160 22697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:12.063480 24114 raft_consensus.cc:740] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:12.065394 24114 consensus_queue.cc:260] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.066864 24114 raft_consensus.cc:3060] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.069512 24104 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:12.076311 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:12.078061 24079 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.079053 24031 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:12.050246 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 49146.8us
I20251026 14:59:12.061710 22731 rpcz_store.cc:276] 1026 14:59:07.208700 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.450329 (+4241629us) service_pool.cc:225] Handling call
1026 14:59:11.818710 (+368381us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:11.818974 (+ 264us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:11.819019 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:11.819099 (+ 80us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:11.819136 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:11.819189 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:12.015388 (+196199us) spinlock_profiling.cc:243] Waited 368 ms on lock 0x7b54000b90d0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:12.015530 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":846042880}
I20251026 14:59:12.082253 24148 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:12.084781 24173 consensus_queue.cc:1048] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251026 14:59:12.085353 22804 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
W20251026 14:59:12.030946 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:12.064998 24228 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:12.085687 24099 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.086392 23951 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:12.062759 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 75783.4us
I20251026 14:59:12.086766 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.087967 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:12.090132 22799 raft_consensus.cc:1275] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
W20251026 14:59:12.090381 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:12.090821 24032 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.091972 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.096220 24073 raft_consensus.cc:1027] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: attempt to promote peer 248493ec312e4398b828c6cb00e5c4b1: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20251026 14:59:12.107453 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:12.105701 22806 raft_consensus.cc:2468] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:12.111404 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:12.113919 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:12.114414 22808 raft_consensus.cc:2468] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:12.109302 22799 pending_rounds.cc:85] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821: Aborting all ops after (but not including) 2
W20251026 14:59:12.117111 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:12.120946 22801 raft_consensus.cc:2410] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:12.122351 22807 raft_consensus.cc:1240] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:12.122996 22805 raft_consensus.cc:1240] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:12.118839 22799 pending_rounds.cc:107] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20251026 14:59:12.110530 24108 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.131001 22802 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.136312 22700 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:12.142565 22554 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38884 (request call id 155) took 8176 ms. Trace:
I20251026 14:59:12.142997 22697 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:12.134064 22799 raft_consensus.cc:2889] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:12.124421 24125 consensus_queue.cc:1048] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.017s
I20251026 14:59:12.123813 24222 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:12.152290 22806 raft_consensus.cc:2410] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251026 14:59:12.163581 22725 raft_consensus.cc:1240] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:12.138206 24100 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.169773 22697 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:12.162264 24173 raft_consensus.cc:2749] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:12.164055 24203 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.158690 24224 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:12.176914 24206 raft_consensus.cc:2749] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:12.183966 22805 raft_consensus.cc:3055] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:12.169150 23370 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:12.184358 22805 raft_consensus.cc:740] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:12.184607 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:12.185320 22804 raft_consensus.cc:3060] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.185173 22805 consensus_queue.cc:260] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:59:12.186375 24060 raft_consensus.cc:2804] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:12.186611 22805 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.186743 24060 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:12.187147 24060 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:12.188865 22697 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:12.189699 24206 raft_consensus.cc:2804] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:12.190152 24206 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:12.190558 24206 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.189708 22801 raft_consensus.cc:1275] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:12.194635 22697 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:12.195700 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.197219 24173 raft_consensus.cc:2749] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:12.196635 22808 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:12.143324 22554 rpcz_store.cc:276] 1026 14:59:03.925133 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:03.926667 (+ 1534us) service_pool.cc:225] Handling call
1026 14:59:04.968490 (+1041823us) spinlock_profiling.cc:243] Waited 29.2 ms on lock 0x7b54003af554. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c7fe0 00007fd8d1166aca 00007fd8d6ed7bbc 00007fd8d6fd1e7e 00007fd8d54d24a4 00007fd8d54d2432 00007fd8d54d2361 00007fd8d54d22dc 00007fd8d54d1592 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:12.101142 (+7132652us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 14:59:06.045786 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 14:59:06.046597 (+ 811us) write_op.cc:432] Acquiring schema lock in shared mode
1026 14:59:06.046648 (+ 51us) write_op.cc:435] Acquired schema lock
1026 14:59:06.046676 (+ 28us) tablet.cc:660] Decoding operations
1026 14:59:06.047849 (+ 1173us) write_op.cc:620] Acquiring the partition lock for write op
1026 14:59:06.047957 (+ 108us) write_op.cc:641] Partition lock acquired for write op
1026 14:59:06.047994 (+ 37us) tablet.cc:683] Acquiring locks for 7 operations
1026 14:59:06.048838 (+ 844us) tablet.cc:699] Row locks acquired
1026 14:59:06.048868 (+ 30us) write_op.cc:260] PREPARE: finished
1026 14:59:06.049068 (+ 200us) write_op.cc:270] Start()
1026 14:59:06.049226 (+ 158us) write_op.cc:276] Timestamp: P: 1761490746049009 usec, L: 0
1026 14:59:06.049257 (+ 31us) op_driver.cc:348] REPLICATION: starting
1026 14:59:06.186144 (+136887us) log.cc:844] Serialized 2801 byte log entry
1026 14:59:06.597264 (+411120us) op_driver.cc:464] REPLICATION: finished
1026 14:59:07.143659 (+546395us) write_op.cc:301] APPLY: starting
1026 14:59:07.155834 (+ 12175us) tablet.cc:1365] starting BulkCheckPresence
1026 14:59:07.210983 (+ 55149us) tablet.cc:1368] finished BulkCheckPresence
1026 14:59:07.214679 (+ 3696us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 14:59:07.268009 (+ 53330us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 14:59:07.302072 (+ 34063us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
1026 14:59:07.309999 (+ 7927us) write_op.cc:312] APPLY: finished
1026 14:59:07.312927 (+ 2928us) log.cc:844] Serialized 58 byte log entry
1026 14:59:07.313105 (+ 178us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:07.313224 (+ 119us) write_op.cc:454] Released schema lock
1026 14:59:07.313290 (+ 66us) write_op.cc:341] FINISH: Updating metrics
1026 14:59:07.313664 (+ 374us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:07.313696 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":624859,"reported_tablets":31,"spinlock_wait_cycles":67094784,"child_traces":[["op",{"apply.queue_time_us":524483,"apply.run_cpu_time_us":4550,"apply.run_wall_time_us":185064,"num_ops":7,"prepare.queue_time_us":218621,"prepare.run_cpu_time_us":53754,"prepare.run_wall_time_us":297707,"raft.queue_time_us":146055,"raft.run_cpu_time_us":21636,"raft.run_wall_time_us":408062,"replication_time_us":547843,"thread_start_us":689818,"threads_started":4,"wal-append.queue_time_us":196180,"wal-append.run_cpu_time_us":5867,"wal-append.run_wall_time_us":1936231}]]}
I20251026 14:59:12.200760 22697 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:12.200872 22804 raft_consensus.cc:2468] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:12.209183 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:12.211787 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 79058.3us
I20251026 14:59:12.214718 22805 raft_consensus.cc:1275] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20251026 14:59:12.215420 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:12.216292 22804 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.217738 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47996 (request call id 79) took 1340 ms. Trace:
I20251026 14:59:12.218115 22725 rpcz_store.cc:276] 1026 14:59:10.877734 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:11.852632 (+974898us) service_pool.cc:225] Handling call
1026 14:59:11.852986 (+ 354us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:12.164150 (+311164us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:12.217523 (+ 53373us) spinlock_profiling.cc:243] Waited 310 ms on lock 0x7b54001ea5d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:12.217688 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":713832320}
I20251026 14:59:12.182050 24031 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:12.215118 22805 pending_rounds.cc:85] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821: Aborting all ops after (but not including) 2
I20251026 14:59:12.208423 23951 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:12.222333 24142 raft_consensus.cc:2804] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:12.227427 24142 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:12.227839 24142 raft_consensus.cc:3060] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:12.225489 22727 raft_consensus.cc:1240] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:12.230090 24225 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.232296 24225 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:12.233049 24108 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:12.234452 24222 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:12.236356 24224 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:12.223210 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:12.241539 24032 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:12.242945 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.243878 22652 raft_consensus.cc:2393] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:12.241547 22697 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:12.248502 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.248735 22697 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:12.249383 22650 raft_consensus.cc:2393] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:12.249451 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.250481 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.249498 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.250674 24224 raft_consensus.cc:2804] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:59:12.255774 22803 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:12.257992 22805 pending_rounds.cc:107] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20251026 14:59:12.258580 22658 raft_consensus.cc:2393] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:12.259047 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.264050 22654 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
W20251026 14:59:12.264410 24270 log.cc:927] Time spent T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20251026 14:59:12.264041 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.266860 24099 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:12.267580 24079 consensus_queue.cc:1059] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:59:12.268682 23951 raft_consensus.cc:3055] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:12.259197 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.269038 23951 raft_consensus.cc:740] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:12.250532 24222 raft_consensus.cc:2804] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:12.269869 24222 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:12.270346 24222 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:12.272889 22805 raft_consensus.cc:2889] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251026 14:59:12.273950 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:12.256237 24224 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:12.276791 24100 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:12.277062 24224 raft_consensus.cc:3060] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:59:12.260671 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.278442 22552 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50170 (request call id 58) took 6538 ms. Trace:
I20251026 14:59:12.259254 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.259670 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.269992 22650 raft_consensus.cc:2410] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:12.281855 24032 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:12.279822 22655 raft_consensus.cc:2393] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
W20251026 14:59:12.282579 22622 connection.cc:232] RPC call timeout handler was delayed by -1.7757446370000025s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1859s for a response
I20251026 14:59:12.285326 24274 consensus_queue.cc:1059] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251026 14:59:12.282425 24032 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.289105 22650 pending_rounds.cc:85] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29: Aborting all ops after (but not including) 5
I20251026 14:59:12.295018 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.295828 22805 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:12.296802 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:12.270359 23951 consensus_queue.cc:260] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.296864 22650 pending_rounds.cc:107] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20251026 14:59:12.301105 22650 raft_consensus.cc:2889] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:12.298373 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.297588 22800 raft_consensus.cc:2376] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3. Re-sending same reply.
I20251026 14:59:12.278848 22552 rpcz_store.cc:276] 1026 14:59:05.794090 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:05.794483 (+ 393us) service_pool.cc:225] Handling call
1026 14:59:12.278231 (+6483748us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 14:59:11.751897 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 14:59:11.752831 (+ 934us) write_op.cc:432] Acquiring schema lock in shared mode
1026 14:59:11.752883 (+ 52us) write_op.cc:435] Acquired schema lock
1026 14:59:11.752914 (+ 31us) tablet.cc:660] Decoding operations
1026 14:59:11.753483 (+ 569us) write_op.cc:620] Acquiring the partition lock for write op
1026 14:59:11.753590 (+ 107us) write_op.cc:641] Partition lock acquired for write op
1026 14:59:11.753632 (+ 42us) tablet.cc:683] Acquiring locks for 1 operations
1026 14:59:11.810793 (+ 57161us) tablet.cc:699] Row locks acquired
1026 14:59:11.823241 (+ 12448us) write_op.cc:260] PREPARE: finished
1026 14:59:11.839970 (+ 16729us) write_op.cc:270] Start()
1026 14:59:11.840142 (+ 172us) write_op.cc:276] Timestamp: P: 1761490751839890 usec, L: 0
1026 14:59:11.840179 (+ 37us) op_driver.cc:348] REPLICATION: starting
1026 14:59:11.841137 (+ 958us) log.cc:844] Serialized 526 byte log entry
1026 14:59:11.976658 (+135521us) op_driver.cc:464] REPLICATION: finished
1026 14:59:12.145501 (+168843us) write_op.cc:301] APPLY: starting
1026 14:59:12.145618 (+ 117us) tablet.cc:1365] starting BulkCheckPresence
1026 14:59:12.150838 (+ 5220us) tablet.cc:1368] finished BulkCheckPresence
1026 14:59:12.170855 (+ 20017us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 14:59:12.255674 (+ 84819us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 14:59:12.273841 (+ 18167us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1026 14:59:12.273911 (+ 70us) write_op.cc:312] APPLY: finished
1026 14:59:12.274161 (+ 250us) log.cc:844] Serialized 22 byte log entry
1026 14:59:12.274366 (+ 205us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:12.274455 (+ 89us) write_op.cc:454] Released schema lock
1026 14:59:12.274510 (+ 55us) write_op.cc:341] FINISH: Updating metrics
1026 14:59:12.274905 (+ 395us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:12.274938 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1383550,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":155565,"apply.run_cpu_time_us":2460,"apply.run_wall_time_us":131406,"num_ops":1,"prepare.queue_time_us":819885,"prepare.run_cpu_time_us":26482,"prepare.run_wall_time_us":193659,"raft.queue_time_us":24740,"raft.run_cpu_time_us":6943,"raft.run_wall_time_us":28231,"replication_time_us":136327,"thread_start_us":582363,"threads_started":4,"wal-append.queue_time_us":108123}]]}
I20251026 14:59:12.299103 23951 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.309265 24032 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:12.285495 24099 consensus_queue.cc:1059] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20251026 14:59:12.314026 22804 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:12.317673 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.318270 22800 raft_consensus.cc:2393] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:12.309635 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:12.288175 24222 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.297194 24203 raft_consensus.cc:2749] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:12.313675 24060 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:12.327425 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.328569 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.329703 24258 raft_consensus.cc:3055] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:12.330099 24258 raft_consensus.cc:740] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:12.330971 24258 consensus_queue.cc:260] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:12.332515 24258 raft_consensus.cc:3060] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:12.333993 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.334429 22802 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:12.325906 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.321044 24142 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.337713 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.338550 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:12.339149 22652 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:12.339533 23917 consensus_queue.cc:1059] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251026 14:59:12.340682 23917 raft_consensus.cc:3055] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:12.340989 23917 raft_consensus.cc:740] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:12.343175 24115 raft_consensus.cc:3055] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:12.352345 24115 raft_consensus.cc:740] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:12.353044 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.354151 23399 raft_consensus.cc:2393] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 6: Already voted for candidate 248493ec312e4398b828c6cb00e5c4b1 in this term.
I20251026 14:59:12.353693 23917 consensus_queue.cc:260] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:12.355191 23917 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:12.355782 23400 raft_consensus.cc:1240] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:12.355996 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.357097 22802 raft_consensus.cc:2410] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 4 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:12.298010 24206 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.361311 24222 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:12.362532 24206 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:12.362723 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.343760 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.363461 22657 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:12.364024 22655 raft_consensus.cc:2393] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:12.325168 22799 raft_consensus.cc:2376] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5. Re-sending same reply.
I20251026 14:59:12.343786 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.348862 24100 consensus_queue.cc:1059] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:12.357796 24079 consensus_queue.cc:1059] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251026 14:59:12.338991 24142 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:12.359128 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.367659 22658 raft_consensus.cc:2468] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:59:12.370560 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.374739 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.375931 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.328606 24060 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:12.377239 22657 raft_consensus.cc:2468] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:12.384159 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.386216 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.386045 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.386931 22653 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:12.374845 24099 raft_consensus.cc:3055] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:12.389884 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.390734 22807 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:12.389760 24099 raft_consensus.cc:740] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:12.387763 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.394536 22799 raft_consensus.cc:2410] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:12.394273 24099 consensus_queue.cc:260] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:12.397171 22658 raft_consensus.cc:1275] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251026 14:59:12.374701 24108 consensus_queue.cc:1048] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20251026 14:59:12.398766 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:59:12.400979 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:12.401451 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:12.402776 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.406004 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:12.406430 22805 pending_rounds.cc:85] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821: Aborting all ops after (but not including) 1
I20251026 14:59:12.406699 22805 pending_rounds.cc:107] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20251026 14:59:12.406926 22805 raft_consensus.cc:2889] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:12.407164 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:12.407840 22651 raft_consensus.cc:2393] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:12.407338 22807 raft_consensus.cc:2393] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 4: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:12.408746 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.409708 22800 raft_consensus.cc:2393] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:12.398031 24142 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:12.410585 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:12.410881 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.410740 24115 consensus_queue.cc:260] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.411465 22658 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:12.411690 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.408802 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.412828 22807 raft_consensus.cc:2393] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:12.412712 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.413509 22804 raft_consensus.cc:2393] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:12.413666 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.414237 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:12.414754 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.415436 22807 raft_consensus.cc:2393] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:12.415642 22804 raft_consensus.cc:1275] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251026 14:59:12.416365 24099 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:12.409555 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.417079 22657 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
W20251026 14:59:12.420578 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 80476.9us
I20251026 14:59:12.412144 24115 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.408663 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.419857 22652 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 6.
I20251026 14:59:12.430891 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:12.425408 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:12.426503 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.432746 22801 raft_consensus.cc:2410] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:12.416343 22807 raft_consensus.cc:1240] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:12.448496 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.452333 22652 raft_consensus.cc:1240] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20251026 14:59:12.459043 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:12.461843 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:12.462823 22804 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.461822 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.466826 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.400046 24224 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.474962 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.478547 24224 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:12.410610 24142 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.480670 24142 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:12.481040 22657 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:12.484099 23370 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:12.484778 24032 consensus_queue.cc:1048] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:59:12.485348 24274 raft_consensus.cc:2804] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:12.485910 24274 raft_consensus.cc:697] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 LEADER]: Becoming Leader. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:12.486909 24274 consensus_queue.cc:237] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:12.488741 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:12.489914 22650 raft_consensus.cc:1240] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20251026 14:59:12.491006 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.491602 22650 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:12.492440 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.494118 22801 raft_consensus.cc:1275] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20251026 14:59:12.494966 22659 raft_consensus.cc:1240] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:12.495419 22697 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:12.496228 22659 raft_consensus.cc:1240] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:12.498157 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:12.498430 24060 consensus_queue.cc:1048] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251026 14:59:12.498939 22727 raft_consensus.cc:1275] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:59:12.489092 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.499764 22655 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:12.457885 24237 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:12.500557 24237 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.501349 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:12.501933 22725 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:12.502074 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.502604 24237 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:12.503068 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:12.506018 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:12.506912 22727 raft_consensus.cc:2393] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 4: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:12.508291 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.509204 22656 raft_consensus.cc:2393] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:12.508347 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:12.509744 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:12.510043 22725 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:12.511813 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:12.512382 22731 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.517726 22727 raft_consensus.cc:1275] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:12.518035 24206 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:12.518621 24206 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.519675 24142 raft_consensus.cc:2804] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:12.520074 24142 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:12.520438 24142 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.518332 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.520717 24206 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:12.524691 22654 raft_consensus.cc:2393] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
W20251026 14:59:12.493194 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:12.526048 22653 raft_consensus.cc:1240] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20251026 14:59:12.527364 22697 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:12.528654 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:12.529322 22656 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 5
I20251026 14:59:12.531529 22653 raft_consensus.cc:1240] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:12.532473 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.533675 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.530273 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.528698 22697 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:12.536952 24206 raft_consensus.cc:2804] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:12.537204 22697 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:12.537495 24206 raft_consensus.cc:697] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:12.538053 24222 raft_consensus.cc:3060] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:12.538420 24206 consensus_queue.cc:237] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.538877 22697 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:12.534474 22655 raft_consensus.cc:2368] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4.
I20251026 14:59:12.540560 24125 raft_consensus.cc:2749] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:12.541205 24125 raft_consensus.cc:2749] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251026 14:59:12.541388 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:12.544847 22697 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:12.546656 22656 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:12.548352 22731 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:12.549165 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.549186 24237 raft_consensus.cc:2749] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:59:12.549507 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.550370 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:12.550699 22697 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:12.551753 24222 raft_consensus.cc:2749] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:12.552016 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:12.560499 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:12.612722 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:12.566452 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.567910 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:12.565970 22800 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:12.572114 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:12.569988 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.567214 23400 raft_consensus.cc:2468] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:12.567299 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.584214 22808 raft_consensus.cc:1240] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:12.570282 23399 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:12.615083 22650 raft_consensus.cc:2376] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5. Re-sending same reply.
I20251026 14:59:12.572175 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:12.602721 22659 raft_consensus.cc:2393] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:12.614284 22657 raft_consensus.cc:2410] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251026 14:59:12.616086 24237 raft_consensus.cc:2749] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251026 14:59:12.616279 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 65692.1us
I20251026 14:59:12.617588 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:12.619077 22725 raft_consensus.cc:2468] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:12.617719 22801 raft_consensus.cc:3060] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:59:12.617946 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:12.618530 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.630988 22697 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:12.631489 22800 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:12.536226 22799 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 6.
I20251026 14:59:12.620961 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.634130 22734 raft_consensus.cc:2376] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4. Re-sending same reply.
I20251026 14:59:12.635138 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:12.617231 22802 raft_consensus.cc:2410] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:12.636031 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.636750 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.639521 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:12.641347 22731 raft_consensus.cc:2393] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 4: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:12.643323 22697 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:12.645061 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:12.647001 22650 raft_consensus.cc:2393] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:12.648905 22697 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:12.649823 22808 raft_consensus.cc:2393] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 4: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:12.651569 22657 raft_consensus.cc:1240] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:12.651367 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:12.652941 22800 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.654932 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.661303 22697 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:12.662942 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:12.664139 22725 raft_consensus.cc:2410] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251026 14:59:12.667487 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:12.672299 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:12.673154 22808 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.667821 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:12.675802 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:12.676527 22734 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:12.667642 24203 raft_consensus.cc:2804] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:12.677645 24203 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:12.678030 24203 raft_consensus.cc:3060] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:12.667498 24237 raft_consensus.cc:2749] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:12.684059 22659 raft_consensus.cc:2368] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4.
I20251026 14:59:12.683966 22804 raft_consensus.cc:2468] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:12.682780 24125 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.699700 22697 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:12.701977 22622 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:12.717971 22801 raft_consensus.cc:2468] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:59:12.719240 22804 raft_consensus.cc:1240] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
I20251026 14:59:12.723954 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:12.724584 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:12.725659 22801 raft_consensus.cc:1240] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4. Ops: []
W20251026 14:59:12.727425 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:12.729641 22725 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:12.731405 22734 raft_consensus.cc:2393] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:12.731251 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:12.732039 22725 raft_consensus.cc:2376] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4. Re-sending same reply.
I20251026 14:59:12.734344 22697 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:12.716502 24237 raft_consensus.cc:2749] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:12.729642 24153 raft_consensus.cc:2749] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:12.737180 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:12.742048 22801 raft_consensus.cc:2410] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20251026 14:59:12.742223 22774 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:12.747737 22800 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:12.733255 24222 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:12.743657 22552 catalog_manager.cc:5649] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:12.758041 22774 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:12.737010 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:12.761811 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:12.762930 22552 catalog_manager.cc:5649] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 2 to 3, leader changed from df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 3 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:12.764418 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:12.764717 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:12.781263 22732 raft_consensus.cc:2393] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:12.765931 22730 raft_consensus.cc:1275] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20251026 14:59:12.781837 22729 raft_consensus.cc:2410] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251026 14:59:12.766912 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:12.752434 24225 consensus_queue.cc:1048] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20251026 14:59:12.757529 24115 raft_consensus.cc:2804] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:12.759836 24237 consensus_queue.cc:1059] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20251026 14:59:12.775528 22805 raft_consensus.cc:1275] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20251026 14:59:12.780974 22725 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:12.766230 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:12.765626 22728 raft_consensus.cc:1275] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:59:12.782871 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:12.747435 24224 raft_consensus.cc:2749] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:12.785012 22805 pending_rounds.cc:85] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821: Aborting all ops after (but not including) 1
I20251026 14:59:12.794708 22805 pending_rounds.cc:107] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20251026 14:59:12.795019 22805 raft_consensus.cc:2889] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:12.728580 22659 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55"
dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:46598
I20251026 14:59:12.795941 22659 raft_consensus.cc:606] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:12.796397 22659 raft_consensus.cc:614] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251026 14:59:12.785418 22622 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:12.801942 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:12.813779 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:12.785406 22802 raft_consensus.cc:3060] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.817106 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:12.817674 22552 catalog_manager.cc:5649] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:59:12.786906 24115 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:12.798652 24099 raft_consensus.cc:2804] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:12.838845 24099 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:12.839365 24099 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:12.844288 22808 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:12.828299 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:12.834736 24288 log.cc:927] Time spent T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20251026 14:59:12.844455 22731 raft_consensus.cc:2468] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:12.818190 22774 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:12.855588 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 70295.2us
I20251026 14:59:12.863969 24032 raft_consensus.cc:2804] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:12.842576 24115 raft_consensus.cc:3060] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:12.873840 24224 raft_consensus.cc:3055] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:12.877081 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:12.824880 24142 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.877708 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:12.900187 22733 raft_consensus.cc:2468] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:12.878710 22734 raft_consensus.cc:2393] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 4: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:12.878597 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:12.901226 22729 raft_consensus.cc:3060] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.881719 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:12.881912 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:12.883810 24203 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.885684 22552 catalog_manager.cc:5649] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 1 to 3, leader changed from df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 3 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
W20251026 14:59:12.888633 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 74166.3us
I20251026 14:59:12.902297 22732 raft_consensus.cc:2376] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4. Re-sending same reply.
I20251026 14:59:12.898403 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:12.897961 24032 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:12.903112 22726 raft_consensus.cc:2393] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 4: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:12.890709 22730 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:12.905392 22725 raft_consensus.cc:2376] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4. Re-sending same reply.
I20251026 14:59:12.897845 24079 raft_consensus.cc:2804] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:12.907648 24079 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:12.906474 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:12.908257 22622 leader_election.cc:400] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5.
I20251026 14:59:12.908573 22622 leader_election.cc:403] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20251026 14:59:12.908412 22733 raft_consensus.cc:3060] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.899010 24125 raft_consensus.cc:2749] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:12.898238 24224 raft_consensus.cc:740] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:12.913111 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:12.909690 24079 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.913352 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:12.917083 22730 raft_consensus.cc:2410] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20251026 14:59:12.914625 22622 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:12.913673 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:12.920197 22729 raft_consensus.cc:2468] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:12.914517 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:12.912250 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:12.920969 22732 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.921051 22734 raft_consensus.cc:2376] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3. Re-sending same reply.
I20251026 14:59:12.899809 22728 raft_consensus.cc:2468] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:59:12.908116 24032 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:12.928768 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:12.929757 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:12.930402 22726 raft_consensus.cc:2410] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:12.928627 22622 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:12.933043 22730 raft_consensus.cc:1240] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:12.917392 24108 raft_consensus.cc:2749] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5.
I20251026 14:59:12.934919 22774 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:12.923120 24142 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:12.931787 24104 raft_consensus.cc:2804] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
W20251026 14:59:12.941483 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:12.939464 24103 raft_consensus.cc:2804] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:12.938124 23951 raft_consensus.cc:2749] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:12.922290 22552 catalog_manager.cc:5649] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 1 to 3, now has a pending config: VOTER df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1), VOTER 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2), VOTER 0d684445287b450a8f738b6ad4b59821 (127.21.2.3), VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4). New cstate: current_term: 3 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:59:12.948053 22726 raft_consensus.cc:1240] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
I20251026 14:59:12.942818 24103 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:12.949901 22730 raft_consensus.cc:1240] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4. Ops: []
W20251026 14:59:12.951634 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 205090us
I20251026 14:59:12.954658 22774 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:12.956074 22697 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:12.956534 24141 consensus_queue.cc:1059] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:12.949137 24103 raft_consensus.cc:3060] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.952419 24104 raft_consensus.cc:697] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:12.960737 24153 consensus_queue.cc:1059] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:59:12.961565 24104 consensus_queue.cc:237] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.961998 24141 raft_consensus.cc:3055] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:12.962538 24141 raft_consensus.cc:740] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:12.963572 24141 consensus_queue.cc:260] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.965827 24153 raft_consensus.cc:3055] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:12.966406 24153 raft_consensus.cc:740] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:12.967944 22806 raft_consensus.cc:1275] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251026 14:59:12.967494 24153 consensus_queue.cc:260] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:12.965258 24141 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:12.969764 23917 consensus_queue.cc:1048] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
I20251026 14:59:12.959321 22622 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:12.985579 22726 raft_consensus.cc:1275] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:12.984884 22552 catalog_manager.cc:5649] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 3 to 4, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 4 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:12.987216 22552 catalog_manager.cc:5649] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 1 to 4, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 4 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:12.996887 22730 raft_consensus.cc:1275] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:59:12.990662 22697 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:13.008203 24125 raft_consensus.cc:2804] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:13.004361 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.008858 24125 raft_consensus.cc:697] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
W20251026 14:59:13.010885 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 51852us
I20251026 14:59:12.969172 24153 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:12.981829 24142 raft_consensus.cc:2749] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:59:13.016788 22726 raft_consensus.cc:1240] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:13.014701 22552 catalog_manager.cc:5649] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
W20251026 14:59:13.022109 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:13.026105 24032 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.028452 24032 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:13.030155 22774 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:13.031832 22697 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:12.946998 24224 consensus_queue.cc:260] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.032699 22733 raft_consensus.cc:2468] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:13.010952 24228 raft_consensus.cc:2804] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:13.000574 23951 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:13.013652 22728 raft_consensus.cc:1275] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:13.016495 24222 raft_consensus.cc:2749] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:13.035429 22697 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:13.024303 22552 catalog_manager.cc:5649] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:59:13.042822 22622 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:13.046463 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.047699 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:13.021634 24104 raft_consensus.cc:2804] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:13.060211 22552 catalog_manager.cc:5649] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 1 to 4, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 4 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:13.062626 22552 catalog_manager.cc:5649] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 2 to 3, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 3 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:13.066293 22651 raft_consensus.cc:1275] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251026 14:59:13.035823 24203 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:13.044734 24228 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:13.010217 24125 consensus_queue.cc:237] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.046985 24237 raft_consensus.cc:2804] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:13.075042 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.076386 24228 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:13.061226 24104 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:13.046717 24031 consensus_queue.cc:1048] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20251026 14:59:13.085803 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.073525 24303 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:13.042363 24032 consensus_queue.cc:1048] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20251026 14:59:13.061614 23951 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.093154 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.083750 24104 raft_consensus.cc:3060] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:13.105664 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:13.108385 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 78373.8us
I20251026 14:59:13.072620 24099 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.106689 22726 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:13.092587 24108 raft_consensus.cc:2804] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:13.090111 24303 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:13.110239 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 74896.5us
I20251026 14:59:13.110208 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:13.111323 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 68625.6us
I20251026 14:59:13.114010 22733 raft_consensus.cc:2393] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:13.114084 22622 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:13.114850 24307 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:13.112643 22774 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:13.115211 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.116587 22733 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:13.117735 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.118451 22733 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:13.118618 24313 raft_consensus.cc:2749] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:13.119606 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.119896 22774 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:13.120558 24313 raft_consensus.cc:2749] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:13.120594 22733 raft_consensus.cc:2393] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 2: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:13.121670 22774 leader_election.cc:400] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4.
I20251026 14:59:13.122327 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.123688 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:13.124665 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:13.122328 22774 leader_election.cc:403] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20251026 14:59:13.124542 22733 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:13.090137 24274 consensus_queue.cc:1048] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20251026 14:59:13.110087 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.130262 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.112214 24108 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:13.154958 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:13.160503 22774 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:13.162015 24313 raft_consensus.cc:2749] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4.
I20251026 14:59:13.162736 24313 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:13.163146 24313 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.164029 24099 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:13.164654 24099 raft_consensus.cc:2804] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:13.115504 24307 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.124465 24303 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
W20251026 14:59:13.171366 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:13.165640 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.188855 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:13.153698 24031 raft_consensus.cc:2804] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:13.182173 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:13.182825 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.183423 24307 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:13.183601 24274 raft_consensus.cc:493] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:13.165349 24108 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:13.167917 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:13.193166 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:13.193265 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:13.184144 22733 raft_consensus.cc:2376] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4. Re-sending same reply.
I20251026 14:59:13.185838 24103 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.189070 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:13.188961 22773 leader_election.cc:304] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:13.190137 22732 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:13.196266 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.190567 22726 raft_consensus.cc:2393] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:13.165115 24099 raft_consensus.cc:697] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:13.204561 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:13.205587 22697 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:13.206290 22656 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
W20251026 14:59:13.206713 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 91284.7us
I20251026 14:59:13.207052 22659 raft_consensus.cc:2393] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:13.165138 24313 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:13.201198 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:13.207931 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:13.208132 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:13.208236 22624 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:13.208425 22700 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:13.211174 22729 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:13.212082 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.202345 24274 raft_consensus.cc:515] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:13.222287 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:13.224251 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:13.170466 22728 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:13.224666 22650 raft_consensus.cc:2368] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5.
I20251026 14:59:13.215569 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.225906 22808 raft_consensus.cc:2410] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20251026 14:59:13.228363 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:13.228991 22807 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
W20251026 14:59:13.229532 24302 log.cc:927] Time spent T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.103s user 0.001s sys 0.000s
I20251026 14:59:13.231382 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.232491 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.232548 22773 leader_election.cc:304] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:13.233637 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:13.213485 24031 raft_consensus.cc:697] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:13.240140 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:13.244889 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:13.250094 24099 consensus_queue.cc:237] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.253844 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:13.229805 24274 leader_election.cc:290] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:13.263839 22729 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:13.228986 22799 raft_consensus.cc:1275] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251026 14:59:13.261793 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.226660 24307 raft_consensus.cc:2749] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:13.257128 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.267252 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:13.268428 22729 raft_consensus.cc:2368] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4.
I20251026 14:59:13.213820 22727 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:13.270015 22726 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:13.236263 24032 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:13.274030 22773 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:13.267915 22622 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:13.282298 22808 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:13.283551 22773 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:13.252614 24079 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:59:13.307544 22803 raft_consensus.cc:2468] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 3.
I20251026 14:59:13.309026 22773 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:13.304736 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:13.285517 24297 log.cc:927] Time spent T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.201s user 0.000s sys 0.001s
I20251026 14:59:13.265057 24315 raft_consensus.cc:2749] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:13.285887 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.343115 22807 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:13.343564 22805 raft_consensus.cc:1240] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20251026 14:59:13.308826 23951 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:13.310173 24314 raft_consensus.cc:2804] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:13.286573 24032 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:13.258613 24311 consensus_queue.cc:1048] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.085s
I20251026 14:59:13.357468 24311 consensus_queue.cc:1243] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer 0d684445287b450a8f738b6ad4b59821 log is divergent from this leader: its last log entry 1.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251026 14:59:13.317559 24313 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:13.359261 24313 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.321071 24307 raft_consensus.cc:2749] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251026 14:59:13.363842 24237 raft_consensus.cc:697] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:13.367018 24323 raft_consensus.cc:2804] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:13.356420 24314 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:13.360855 24222 raft_consensus.cc:2804] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:13.348860 24031 consensus_queue.cc:237] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:13.378026 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 172516us
I20251026 14:59:13.369366 24315 raft_consensus.cc:2804] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:13.382421 24315 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:13.372211 22728 raft_consensus.cc:2393] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 4: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
W20251026 14:59:13.384408 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 73498.2us
I20251026 14:59:13.383395 24315 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:13.366739 22727 raft_consensus.cc:1240] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
W20251026 14:59:13.389317 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 70020.7us
I20251026 14:59:13.392637 22697 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:13.396344 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 187967us
I20251026 14:59:13.366617 24032 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:13.387399 22622 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:13.399919 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:13.393955 22802 consensus_meta.cc:287] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.536s user 0.010s sys 0.002s
I20251026 14:59:13.386896 24323 raft_consensus.cc:697] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:13.385550 24314 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:13.421247 22700 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:13.423331 24320 raft_consensus.cc:2804] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:13.423475 22700 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:13.412420 22622 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:13.429888 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 120984us
I20251026 14:59:13.431370 22773 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:13.411226 24103 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:13.411697 24206 raft_consensus.cc:2749] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:13.409859 22802 raft_consensus.cc:2468] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:13.411475 24310 raft_consensus.cc:2804] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:13.432989 22700 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:13.444530 23367 leader_election.cc:304] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters:
I20251026 14:59:13.438220 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.453181 22734 raft_consensus.cc:1240] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
W20251026 14:59:13.444073 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 51569.9us
I20251026 14:59:13.415179 24313 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:13.451956 22622 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:13.438374 24310 raft_consensus.cc:697] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:13.457779 24237 consensus_queue.cc:237] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:13.457614 24142 raft_consensus.cc:2749] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251026 14:59:13.459617 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 251432us
W20251026 14:59:13.533731 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 102435us
I20251026 14:59:13.474748 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:13.461777 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:13.541174 22697 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:13.538432 22622 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:13.424099 24303 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:13.544198 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 111266us
I20251026 14:59:13.433487 24308 raft_consensus.cc:2749] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:13.554562 22700 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:13.541180 22726 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:13.417740 24323 consensus_queue.cc:237] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.437438 24222 raft_consensus.cc:697] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:13.553308 22697 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:13.431473 24320 raft_consensus.cc:697] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:13.572276 22700 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:13.537762 24153 raft_consensus.cc:2804] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:13.458184 24312 raft_consensus.cc:2804] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:13.441985 24079 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:13.478567 24258 raft_consensus.cc:2804] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:13.466804 24322 raft_consensus.cc:2749] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:59:13.572481 22624 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:13.562734 24311 raft_consensus.cc:2804] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:13.559006 24125 raft_consensus.cc:2804] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:13.457854 24327 raft_consensus.cc:2804] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:59:13.537400 24206 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:13.443367 24315 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.583246 24294 raft_consensus.cc:2749] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:13.557307 24141 consensus_queue.cc:1059] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20251026 14:59:13.582055 24153 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:13.595446 22700 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:13.474236 24103 consensus_queue.cc:1059] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.113s
I20251026 14:59:13.587936 24258 raft_consensus.cc:493] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:13.556743 24310 consensus_queue.cc:237] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.588424 24313 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:13.588325 24312 raft_consensus.cc:697] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:13.590224 24125 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:13.598556 24115 consensus_meta.cc:287] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.695s user 0.012s sys 0.000s
I20251026 14:59:13.587553 24104 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.599429 24153 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:13.590157 24206 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:13.589138 24031 consensus_queue.cc:1059] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20251026 14:59:13.600605 24258 raft_consensus.cc:3060] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:13.614954 22733 raft_consensus.cc:2468] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
W20251026 14:59:13.616005 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:13.617162 22652 raft_consensus.cc:2368] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 3. Current term is 4.
I20251026 14:59:13.618510 24142 raft_consensus.cc:2804] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:13.619302 24308 raft_consensus.cc:2749] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:13.618835 24222 consensus_queue.cc:237] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:13.619685 24320 consensus_queue.cc:237] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.604238 24313 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:13.603966 24115 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.622613 24224 raft_consensus.cc:3060] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:13.613376 24206 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:13.603188 24125 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:13.623943 22697 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:13.592518 24332 consensus_queue.cc:1048] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.038s
I20251026 14:59:13.593739 24328 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:13.651681 24338 raft_consensus.cc:2749] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:13.651479 24314 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.674197 24327 raft_consensus.cc:697] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 2 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:13.677572 24327 consensus_queue.cc:237] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:13.686321 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:13.686556 24322 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:13.687413 24322 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.688688 24141 raft_consensus.cc:2749] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:13.690258 24337 raft_consensus.cc:3055] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:13.690650 24337 raft_consensus.cc:740] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:13.692740 24322 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:13.694592 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 99243.9us
I20251026 14:59:13.695621 22700 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:13.696620 24344 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:13.697803 22700 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:13.699086 24327 raft_consensus.cc:2804] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:13.700136 24327 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:13.701471 24327 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:13.691614 24337 consensus_queue.cc:260] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.702404 24322 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:13.703213 24337 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:13.703182 24322 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:13.708698 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 136283us
I20251026 14:59:13.710310 22624 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
W20251026 14:59:13.726675 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:13.672837 24228 consensus_meta.cc:287] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.587s user 0.010s sys 0.001s
I20251026 14:59:13.662729 24142 raft_consensus.cc:697] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
W20251026 14:59:13.742806 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 45064.2us
W20251026 14:59:13.743757 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 76702.9us
I20251026 14:59:13.745077 22700 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:13.708105 24303 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:13.672652 24308 raft_consensus.cc:2804] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:13.755676 22700 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:13.741886 24141 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:13.760865 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 186472us
I20251026 14:59:13.738240 24311 raft_consensus.cc:697] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:13.760789 22700 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:13.709523 24294 consensus_queue.cc:1059] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
I20251026 14:59:13.759513 24225 raft_consensus.cc:2749] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:13.750005 24228 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:13.779927 24108 consensus_meta.cc:287] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.566s user 0.011s sys 0.003s
W20251026 14:59:13.781625 22774 leader_election.cc:336] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185): Timed out: RequestConsensusVote RPC to 127.21.2.1:46185 timed out after 1.657s (SENT)
I20251026 14:59:13.761780 24315 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:13.672816 24312 consensus_queue.cc:237] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.797220 22622 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:13.803032 24346 consensus_queue.cc:1048] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:13.804318 24228 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:13.780335 24104 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:13.811666 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 50999.9us
I20251026 14:59:13.812774 22700 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:13.768065 24141 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.708055 24328 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:59:13.812093 22622 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:13.790164 24322 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:13.817229 24330 log.cc:927] Time spent T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.160s user 0.000s sys 0.001s
I20251026 14:59:13.797771 24108 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.829612 24115 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:13.846717 24225 raft_consensus.cc:2749] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:13.847978 24141 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:13.850435 24348 raft_consensus.cc:2804] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:13.850859 24125 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.853740 22658 raft_consensus.cc:1240] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:13.854732 24108 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:13.780324 24333 consensus_queue.cc:1048] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.176s
I20251026 14:59:13.815809 24103 raft_consensus.cc:2804] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:13.817409 24325 raft_consensus.cc:2804] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:13.842310 24104 raft_consensus.cc:2804] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:13.865770 24104 raft_consensus.cc:697] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:13.867225 24104 consensus_queue.cc:237] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.850943 24348 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:13.842072 22622 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:13.848570 24311 consensus_queue.cc:237] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.862426 24032 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.878087 22803 raft_consensus.cc:1240] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5. Ops: []
W20251026 14:59:13.880082 24340 log.cc:927] Time spent T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
W20251026 14:59:13.881752 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:13.863669 24325 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:13.860422 24308 raft_consensus.cc:697] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:13.885125 22774 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:13.869894 24313 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:59:13.890121 24341 raft_consensus.cc:2749] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:13.814847 24142 consensus_queue.cc:237] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.881717 24339 raft_consensus.cc:3055] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:59:13.872453 24103 raft_consensus.cc:697] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:13.897168 22802 raft_consensus.cc:1240] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20251026 14:59:13.898181 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:13.898975 22807 raft_consensus.cc:1240] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251026 14:59:13.873545 24348 raft_consensus.cc:3060] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:13.903529 22806 raft_consensus.cc:1275] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251026 14:59:13.904932 22801 pending_rounds.cc:85] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821: Aborting all ops after (but not including) 1
I20251026 14:59:13.907227 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:13.908633 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:13.908631 22624 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:13.890681 24325 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:13.879453 24314 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:13.919909 22773 leader_election.cc:336] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.688s (SENT)
I20251026 14:59:13.938937 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:13.939350 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:13.939801 22654 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:13.940071 22653 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:13.921941 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:13.909993 24141 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
W20251026 14:59:13.924104 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 111395us
I20251026 14:59:13.927563 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:13.910377 24339 raft_consensus.cc:740] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:13.931819 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:13.922796 22803 raft_consensus.cc:1275] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20251026 14:59:13.913488 24125 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:13.948940 22657 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:13.940975 22804 raft_consensus.cc:3060] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
W20251026 14:59:13.938582 22773 leader_election.cc:336] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.696s (SENT)
I20251026 14:59:13.891630 24222 consensus_queue.cc:1059] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.034s
I20251026 14:59:13.943035 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:13.909806 24079 raft_consensus.cc:2749] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:13.954367 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:13.954995 22800 raft_consensus.cc:3060] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:13.931391 22799 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:13.957482 24350 raft_consensus.cc:2749] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:13.946655 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:13.961606 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:13.962023 24344 raft_consensus.cc:2749] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:13.962342 22654 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:13.963394 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:13.963958 22654 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:13.930003 22801 pending_rounds.cc:107] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 1.2
I20251026 14:59:13.965524 22801 raft_consensus.cc:2917] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Aborting config change with OpId 1.2: Aborted: Op aborted by new leader
I20251026 14:59:13.938592 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:13.966037 22801 raft_consensus.cc:2889] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:13.966624 24328 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:13.966696 22801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:49650 (request call id 4) took 18662 ms. Trace:
I20251026 14:59:13.923810 24258 raft_consensus.cc:515] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:13.967238 22801 rpcz_store.cc:276] 1026 14:58:55.303852 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:58:55.304023 (+ 171us) service_pool.cc:225] Handling call
1026 14:58:55.311931 (+ 7908us) log.cc:844] Serialized 485 byte log entry
1026 14:58:56.084714 (+772783us) spinlock_profiling.cc:243] Waited 3.21 ms on lock 0x7b1800301a80. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce0b1992 00007fd8ce0b3d61 00007fd8ce0b3d19 00007fd8ce0b3ca9 00007fd8ce0b3c71 00007fd8ce0b2f6d 0000000000379254 0000000000379099 00007fd8ce094d1a
1026 14:58:56.183372 (+ 98658us) spinlock_profiling.cc:243] Waited 2.24 ms on lock 0x7b1800301a80. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce026caa 00007fd8d7698a29 00007fd8cdee6c30 00007fd8ce0aea94 00007fd8ce0aea05 00007fd8d10b7a5b 00007fd8d113c7b3 00007fd8d11642e2 00007fd8d61082e9
1026 14:59:13.966597 (+17783225us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":169855,"raft.queue_time_us":632830,"raft.run_cpu_time_us":1797,"raft.run_wall_time_us":3410,"spinlock_wait_cycles":12547584,"thread_start_us":185301,"threads_started":1,"wal-append.queue_time_us":231983,"wal-append.run_cpu_time_us":2068,"wal-append.run_wall_time_us":1050749}
I20251026 14:59:13.969090 24258 leader_election.cc:290] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:13.970309 24339 consensus_queue.cc:260] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:13.971493 24339 raft_consensus.cc:3060] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:59:13.973459 24353 raft_consensus.cc:2804] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:13.974138 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:13.974545 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:13.947883 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:13.962105 22656 raft_consensus.cc:2468] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:13.954370 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:13.948148 22700 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:13.948941 22803 pending_rounds.cc:85] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821: Aborting all ops after (but not including) 1
I20251026 14:59:13.980530 22803 pending_rounds.cc:107] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20251026 14:59:13.980852 22803 raft_consensus.cc:2889] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:13.982420 22700 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:13.983045 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 74518.7us
I20251026 14:59:13.983974 22624 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:13.984923 22800 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:13.986270 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:13.986727 22800 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
W20251026 14:59:13.953613 22773 leader_election.cc:336] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.955s (SENT)
I20251026 14:59:13.987700 22773 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:13.974011 24353 raft_consensus.cc:697] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:13.990984 22729 raft_consensus.cc:3060] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 4
I20251026 14:59:13.988883 22554 catalog_manager.cc:5649] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: term changed from 3 to 5, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4). New cstate: current_term: 5 leader_uuid: "248493ec312e4398b828c6cb00e5c4b1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 14:59:13.999255 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:13.944751 24336 raft_consensus.cc:3055] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:14.002974 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:14.007977 22656 raft_consensus.cc:3060] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:14.004472 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 50935.6us
I20251026 14:59:14.005939 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:14.007567 24336 raft_consensus.cc:740] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:14.019459 24336 consensus_queue.cc:260] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:14.022330 22604 auto_rebalancer.cc:254] failed to send replica move request: Aborted: Op aborted by new leader
W20251026 14:59:14.022967 24357 log.cc:927] Time spent T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
I20251026 14:59:14.025185 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:59:14.025748 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 44883.8us
I20251026 14:59:14.025830 22803 raft_consensus.cc:3060] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:14.028443 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:13.997206 24237 raft_consensus.cc:2804] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:14.030491 22700 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:14.017778 22554 catalog_manager.cc:5649] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: term changed from 1 to 3, now has a pending config: VOTER df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1), VOTER 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2), VOTER 0d684445287b450a8f738b6ad4b59821 (127.21.2.3), NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4). New cstate: current_term: 3 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } } pending_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } attrs { replace: true } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:59:14.035382 22774 leader_election.cc:400] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4.
I20251026 14:59:14.036084 22774 leader_election.cc:403] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20251026 14:59:13.995062 22804 raft_consensus.cc:2468] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
W20251026 14:59:14.032305 22773 leader_election.cc:336] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.691s (SENT)
I20251026 14:59:14.005831 24344 raft_consensus.cc:2749] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:14.032892 22652 raft_consensus.cc:1275] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251026 14:59:14.005991 24314 raft_consensus.cc:2749] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251026 14:59:14.053965 22730 raft_consensus.cc:2468] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:13.975279 24141 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:14.058228 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 74347.1us
I20251026 14:59:14.059070 22624 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:14.010648 24337 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:14.063457 22700 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:14.053197 22800 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:14.065590 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:14.066221 22804 raft_consensus.cc:1240] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20251026 14:59:14.038487 22656 raft_consensus.cc:2468] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
W20251026 14:59:14.037452 22773 leader_election.cc:336] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 2.045s (SENT)
I20251026 14:59:14.032539 24336 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:13.997160 24313 consensus_queue.cc:1059] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20251026 14:59:14.068646 22773 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:14.071606 22807 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:14.075495 22651 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:14.075769 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:14.076507 22808 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:14.077026 24125 raft_consensus.cc:2749] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:14.079092 22700 leader_election.cc:400] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5.
I20251026 14:59:14.080076 22700 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:14.074882 24341 consensus_queue.cc:1048] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20251026 14:59:14.081792 24125 raft_consensus.cc:2749] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:14.072535 24104 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:14.083257 24104 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:14.084391 22700 leader_election.cc:400] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 3. Current term is 4.
I20251026 14:59:14.072629 24320 raft_consensus.cc:2749] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251026 14:59:14.085834 22624 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:14.086062 24353 consensus_queue.cc:237] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.087143 24125 consensus_queue.cc:1059] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:59:14.088303 24350 raft_consensus.cc:3060] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:14.090536 22624 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:14.091040 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:14.093243 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:14.056988 24115 raft_consensus.cc:2749] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4.
I20251026 14:59:14.095575 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:14.098294 22733 raft_consensus.cc:2468] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:14.109064 22652 raft_consensus.cc:1275] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20251026 14:59:14.162791 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:14.162681 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 72261.9us
I20251026 14:59:14.070844 24222 raft_consensus.cc:2749] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:14.100993 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:14.072276 24237 raft_consensus.cc:697] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:14.075496 24344 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:14.097699 24108 consensus_queue.cc:1048] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.080s
I20251026 14:59:14.178604 24237 consensus_queue.cc:237] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.181975 22658 raft_consensus.cc:1275] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:14.183470 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:14.183979 22659 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:14.033001 24103 consensus_queue.cc:237] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.187821 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:14.188329 24141 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:14.026772 24308 consensus_queue.cc:237] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.190891 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:14.191298 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:14.191682 22726 raft_consensus.cc:2393] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 5: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:14.191972 22734 raft_consensus.cc:2468] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:14.192479 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:14.192762 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:14.070236 24153 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.160545 24294 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:14.198818 22657 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:14.200259 22729 raft_consensus.cc:2468] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 4.
I20251026 14:59:14.201040 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:14.201500 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:14.201618 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:14.202083 22729 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:14.205947 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:14.206048 22808 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:14.206494 22734 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:14.208061 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:14.207903 22622 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:14.174978 24032 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:14.211174 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:14.213802 22725 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:14.177662 24104 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:14.180817 24354 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:14.212311 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:14.220867 22732 raft_consensus.cc:2468] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
W20251026 14:59:14.221194 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:14.214509 22624 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:14.215955 22653 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:14.217324 22803 raft_consensus.cc:2468] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 4.
I20251026 14:59:14.161063 24327 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.098810 24320 consensus_queue.cc:1048] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.105s
I20251026 14:59:14.202929 24312 raft_consensus.cc:2749] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251026 14:59:14.232154 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 194723us
W20251026 14:59:14.236790 22773 leader_election.cc:336] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 2.082s (SENT)
W20251026 14:59:14.231196 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 55127us
I20251026 14:59:14.237932 22774 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:14.240347 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:14.240916 22732 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:14.242376 22806 raft_consensus.cc:1240] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
W20251026 14:59:14.246271 22773 leader_election.cc:336] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.863s (SENT)
I20251026 14:59:14.246649 22773 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:14.246515 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:14.212489 24346 consensus_queue.cc:1059] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20251026 14:59:14.203841 24344 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.212704 24332 consensus_queue.cc:1059] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20251026 14:59:14.100368 24352 consensus_queue.cc:1048] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.139s
I20251026 14:59:14.220497 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:14.207793 24294 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.243624 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:14.221714 24353 raft_consensus.cc:2804] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:14.211450 22799 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
W20251026 14:59:14.359051 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:14.247937 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:14.271377 22773 leader_election.cc:336] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 2.129s (SENT)
I20251026 14:59:14.375468 22773 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:14.364305 22804 raft_consensus.cc:2368] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 5. Current term is 6.
I20251026 14:59:14.261425 22774 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:14.287418 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:14.273312 22731 raft_consensus.cc:2393] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 5: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:14.247462 23369 leader_election.cc:304] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:14.265590 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 51116.2us
I20251026 14:59:14.247936 22725 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
W20251026 14:59:14.270790 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:14.232933 24315 raft_consensus.cc:2749] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:14.371539 22659 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:14.371359 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:14.371960 22653 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:14.384246 22624 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:14.232915 24303 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:14.371199 22654 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
W20251026 14:59:14.383574 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:14.253382 24206 consensus_meta.cc:287] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.535s user 0.007s sys 0.003s
W20251026 14:59:14.265897 24359 log.cc:927] Time spent T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
W20251026 14:59:14.386251 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 184193us
I20251026 14:59:14.384048 22728 raft_consensus.cc:2376] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5. Re-sending same reply.
I20251026 14:59:14.242400 24141 raft_consensus.cc:2804] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
W20251026 14:59:14.386806 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 115469us
I20251026 14:59:14.244518 24222 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:14.383687 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:14.256713 24337 raft_consensus.cc:2749] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:14.393591 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:14.389822 22732 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:14.394392 22728 raft_consensus.cc:1275] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251026 14:59:14.394526 22733 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:14.247625 24364 raft_consensus.cc:3055] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:14.260479 24313 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:14.261706 24322 raft_consensus.cc:2749] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:14.269461 24314 raft_consensus.cc:2749] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:14.389714 22773 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:14.272737 24153 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:14.391314 24274 raft_consensus.cc:2804] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:14.387806 24258 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:14.391409 24115 raft_consensus.cc:2749] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:14.395890 24315 raft_consensus.cc:2764] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20251026 14:59:14.396747 24364 raft_consensus.cc:740] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:14.251531 24325 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.417645 22773 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:14.396944 24353 raft_consensus.cc:697] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:14.420996 22733 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:14.242274 24354 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.392026 24303 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.395478 24206 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.410853 24079 raft_consensus.cc:2749] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:14.395797 24222 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.434865 22773 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:14.432415 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 224613us
I20251026 14:59:14.412020 24258 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:14.419585 24141 raft_consensus.cc:697] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:14.439883 22622 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:14.417045 24104 raft_consensus.cc:2749] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251026 14:59:14.443091 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 58904us
I20251026 14:59:14.447116 22624 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:14.444012 22650 raft_consensus.cc:2393] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 5: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:14.405740 24313 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.450955 22726 raft_consensus.cc:2393] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 6: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
W20251026 14:59:14.452922 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 191610us
I20251026 14:59:14.453502 22774 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:14.453593 22773 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:14.454247 22725 raft_consensus.cc:2468] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:14.454314 22774 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:14.438506 24115 raft_consensus.cc:2749] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:59:14.455662 22622 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:14.438244 24345 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:14.445806 24314 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:14.473011 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:14.455696 22734 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:14.452512 24315 raft_consensus.cc:2749] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:14.458465 22624 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:14.457798 22774 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:14.504078 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 46133.7us
W20251026 14:59:14.505993 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 52449us
I20251026 14:59:14.445582 24323 consensus_queue.cc:1048] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
W20251026 14:59:14.510429 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 54468.9us
I20251026 14:59:14.456555 24322 raft_consensus.cc:2749] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:14.457412 24311 raft_consensus.cc:2804] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:14.456777 24339 raft_consensus.cc:2804] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:14.455791 24338 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:14.457480 24079 raft_consensus.cc:2804] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:14.511706 22774 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:14.521319 22622 leader_election.cc:400] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 5. Current term is 6.
I20251026 14:59:14.503330 24237 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:14.521906 22622 leader_election.cc:403] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20251026 14:59:14.508322 24309 raft_consensus.cc:3055] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:14.522760 24224 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:14.523159 24371 log.cc:927] Time spent T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.101s user 0.000s sys 0.001s
I20251026 14:59:14.523638 24322 raft_consensus.cc:2804] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:14.526139 24372 raft_consensus.cc:2804] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:14.526538 24303 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:14.536471 24338 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:14.530509 24373 log.cc:927] Time spent T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.078s user 0.002s sys 0.000s
I20251026 14:59:14.533489 24344 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:14.533408 24315 consensus_queue.cc:1059] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:14.534823 24206 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:14.505927 24332 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:14.545639 24141 consensus_queue.cc:237] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.561630 24332 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:14.562181 24337 raft_consensus.cc:2804] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:14.562645 24337 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:14.563024 24337 raft_consensus.cc:3060] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:14.563903 24332 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:14.508951 24274 raft_consensus.cc:697] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 LEADER]: Becoming Leader. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:14.509954 24115 raft_consensus.cc:2804] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:14.484602 24345 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.457345 24350 raft_consensus.cc:2749] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:14.508265 24314 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.551148 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:14.567443 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 45090.7us
W20251026 14:59:14.552273 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 93902.2us
I20251026 14:59:14.456269 24364 consensus_queue.cc:260] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.567142 24339 raft_consensus.cc:697] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:14.539040 24325 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:14.556186 24311 raft_consensus.cc:697] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:14.539435 24309 raft_consensus.cc:740] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:14.556936 24377 raft_consensus.cc:2749] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:14.558048 24381 consensus_queue.cc:1048] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:14.527840 24313 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:14.572201 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:14.560755 24338 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:14.521106 24079 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:14.579298 24079 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:14.565088 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:14.537860 22774 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:14.583382 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 45597.5us
I20251026 14:59:14.583729 22773 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:14.566279 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 45172.6us
I20251026 14:59:14.535496 24258 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:14.552937 24322 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:14.571357 24354 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:14.561651 24222 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:14.573071 24339 consensus_queue.cc:237] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.573963 24311 consensus_queue.cc:237] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.511209 24353 consensus_queue.cc:237] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.546288 24237 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:14.572075 24348 consensus_meta.cc:287] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.627s user 0.011s sys 0.000s
I20251026 14:59:14.578106 24314 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:14.578146 24336 consensus_meta.cc:287] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.505s user 0.005s sys 0.004s
I20251026 14:59:14.569111 24364 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:14.621404 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:14.585790 22622 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:14.586722 22624 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:14.588109 22697 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:14.582192 24379 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:14.565929 24115 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:14.622114 24348 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:14.589108 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:14.630030 24348 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:14.596890 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:14.578491 24372 raft_consensus.cc:697] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:14.633852 22806 raft_consensus.cc:2410] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:14.598148 22807 raft_consensus.cc:1240] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:14.598441 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:14.598629 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:14.599589 22800 raft_consensus.cc:1275] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:59:14.590634 24315 raft_consensus.cc:2804] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:14.616469 24203 consensus_queue.cc:1048] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20251026 14:59:14.638387 24315 raft_consensus.cc:697] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:14.600381 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:14.639495 22799 raft_consensus.cc:2393] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 6: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:14.639259 24315 consensus_queue.cc:237] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.641000 24348 raft_consensus.cc:2804] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election won for term 5
W20251026 14:59:14.600533 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 356730us
I20251026 14:59:14.596131 24313 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:14.596264 24322 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:14.603389 24337 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.553139 24224 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.611339 22656 raft_consensus.cc:1240] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:14.652447 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:14.612349 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:14.612769 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:14.614877 22773 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:14.619902 24345 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:14.620204 24237 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:14.575292 24346 raft_consensus.cc:2749] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 5. Current term is 6.
I20251026 14:59:14.620852 22658 raft_consensus.cc:1275] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251026 14:59:14.627424 24312 raft_consensus.cc:2804] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:14.627632 22622 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:14.627851 24115 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:14.592916 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:14.630968 24379 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:14.671062 22774 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:14.590857 24377 raft_consensus.cc:2804] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:14.635608 22805 raft_consensus.cc:2410] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20251026 14:59:14.636240 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 48243.1us
I20251026 14:59:14.622243 24274 consensus_queue.cc:237] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:14.616215 24142 consensus_queue.cc:1048] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.022s
I20251026 14:59:14.634099 24372 consensus_queue.cc:237] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.647775 22624 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:14.644441 24336 raft_consensus.cc:2749] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:14.655112 22773 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:14.655413 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:14.656360 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:14.644322 24313 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.668433 22654 raft_consensus.cc:1240] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:14.631567 24079 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.685132 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:14.671137 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:14.674888 22804 raft_consensus.cc:1275] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:14.672088 24303 raft_consensus.cc:2804] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:14.650712 24348 raft_consensus.cc:697] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:14.678771 22801 raft_consensus.cc:1275] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:14.660804 24314 raft_consensus.cc:2804] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:14.680208 22808 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2"
dest_uuid: "0d684445287b450a8f738b6ad4b59821"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:44012
I20251026 14:59:14.694759 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:14.694995 22807 raft_consensus.cc:1275] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:14.695120 22808 raft_consensus.cc:606] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:14.695483 22800 raft_consensus.cc:2368] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5.
I20251026 14:59:14.695580 22808 raft_consensus.cc:614] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251026 14:59:14.672344 24141 raft_consensus.cc:2804] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:14.687850 24382 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:14.686060 22805 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:14.692824 22655 raft_consensus.cc:2393] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 6: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:14.675441 22774 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:14.672849 24312 raft_consensus.cc:697] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:14.702085 24383 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:14.710819 22774 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:14.716053 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:14.717211 22800 raft_consensus.cc:2393] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:14.718509 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:14.694835 24314 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:14.697957 24141 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:14.704650 24303 raft_consensus.cc:697] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:14.691640 24203 consensus_queue.cc:1048] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20251026 14:59:14.730630 24386 consensus_queue.cc:1048] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.007s
I20251026 14:59:14.730692 24312 consensus_queue.cc:237] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.744232 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:14.731432 22624 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:14.746470 22654 raft_consensus.cc:1240] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:14.732049 24103 consensus_queue.cc:1048] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:59:14.748580 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:14.732936 24379 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:14.713390 24328 raft_consensus.cc:2749] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:59:14.689314 24354 consensus_queue.cc:1059] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20251026 14:59:14.751590 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:14.752133 22730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 1101) took 1756 ms (client timeout 1875 ms). Trace:
W20251026 14:59:14.761797 22730 rpcz_store.cc:269] 1026 14:59:12.995594 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:13.002751 (+ 7157us) service_pool.cc:225] Handling call
1026 14:59:14.751926 (+1749175us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400076554. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:14.752057 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2400808320}
W20251026 14:59:14.755048 24376 log.cc:927] Time spent T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.246s user 0.000s sys 0.000s
I20251026 14:59:14.754973 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:14.733489 24323 consensus_queue.cc:1059] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20251026 14:59:14.729056 24314 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:14.756026 24224 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:14.709874 24382 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.757686 24303 consensus_queue.cc:237] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:59:14.758499 24079 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:14.757681 24225 raft_consensus.cc:3055] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:14.733999 24141 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:14.759260 24153 consensus_queue.cc:1048] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.004s
I20251026 14:59:14.728305 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:14.814759 22651 raft_consensus.cc:1240] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:14.720337 24383 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.763357 24381 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:14.818244 24336 raft_consensus.cc:3055] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:14.818856 24337 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:14.820932 24315 raft_consensus.cc:2804] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:14.759599 24311 raft_consensus.cc:2804] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:14.822228 22800 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba"
dest_uuid: "0d684445287b450a8f738b6ad4b59821"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:44012
I20251026 14:59:14.834599 22654 raft_consensus.cc:1240] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:14.865113 22800 raft_consensus.cc:606] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:14.865726 22800 raft_consensus.cc:614] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251026 14:59:14.835567 22622 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:14.824537 24225 raft_consensus.cc:740] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:14.820086 24382 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:14.834172 24377 raft_consensus.cc:697] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:14.863538 24323 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:14.835225 24336 raft_consensus.cc:740] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:59:14.883200 24374 log.cc:927] Time spent T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.298s user 0.000s sys 0.003s
I20251026 14:59:14.899722 22801 raft_consensus.cc:1275] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20251026 14:59:14.902782 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 156189us
I20251026 14:59:14.839279 24381 raft_consensus.cc:697] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:14.866979 24384 raft_consensus.cc:2749] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20251026 14:59:14.906641 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 252476us
I20251026 14:59:14.907594 22800 raft_consensus.cc:1275] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:14.866097 24315 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
W20251026 14:59:14.906358 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 195610us
I20251026 14:59:14.910288 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:14.908555 22773 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:14.830207 24348 consensus_queue.cc:237] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.905673 24353 raft_consensus.cc:2749] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:14.919239 22774 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:14.905961 24313 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:14.922135 24225 consensus_queue.cc:260] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:14.924353 24225 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:14.919605 22773 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:14.912631 24315 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:14.926795 22803 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5"
dest_uuid: "0d684445287b450a8f738b6ad4b59821"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:44012
I20251026 14:59:14.931154 22803 raft_consensus.cc:606] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:14.930186 22773 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:14.916018 24336 consensus_queue.cc:260] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.931386 24328 raft_consensus.cc:2804] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:14.937635 24401 raft_consensus.cc:2804] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:14.938053 24401 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:14.938424 24401 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:14.905381 24323 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.934151 24384 consensus_queue.cc:1048] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20251026 14:59:14.931480 22803 raft_consensus.cc:614] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251026 14:59:14.937446 24313 consensus_queue.cc:1059] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20251026 14:59:14.956854 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:14.963074 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251026 14:59:14.935693 24406 consensus_queue.cc:1059] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20251026 14:59:14.971383 24258 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:14.972209 24258 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:14.973870 24401 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:14.936055 24377 consensus_queue.cc:237] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.937630 24328 raft_consensus.cc:697] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:14.975744 24405 raft_consensus.cc:2804] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:14.948978 24323 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:14.980123 22554 catalog_manager.cc:5649] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "248493ec312e4398b828c6cb00e5c4b1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 14:59:14.975936 24314 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.982039 22554 catalog_manager.cc:5649] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: term changed from 5 to 6, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4). New cstate: current_term: 6 leader_uuid: "248493ec312e4398b828c6cb00e5c4b1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 14:59:14.990352 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:59:14.993855 22804 raft_consensus.cc:1240] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:14.979198 24383 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:14.997593 24311 raft_consensus.cc:697] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:59:15.006176 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 74191us
I20251026 14:59:14.985906 24322 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.014506 24401 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:59:15.014688 24322 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:14.954618 24381 consensus_queue.cc:237] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.014719 24115 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.992146 24079 raft_consensus.cc:2804] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:15.016862 24115 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:15.010617 24406 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:15.017941 24406 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:14.990419 24405 raft_consensus.cc:697] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
W20251026 14:59:15.019392 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 68584.7us
W20251026 14:59:15.024839 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 94764.4us
I20251026 14:59:15.026651 22773 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:15.029155 24405 consensus_queue.cc:237] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.030181 24115 raft_consensus.cc:2804] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:15.030777 24115 raft_consensus.cc:697] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:15.031688 24115 consensus_queue.cc:237] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.028267 22773 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:15.046409 24320 raft_consensus.cc:3055] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:15.046896 24320 raft_consensus.cc:740] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:15.047956 24258 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:15.047751 24320 consensus_queue.cc:260] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.049925 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:15.054795 24311 consensus_queue.cc:237] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.060626 24314 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:15.062500 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:15.063231 22726 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 4.
W20251026 14:59:15.070745 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:15.070967 22622 leader_election.cc:400] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5.
I20251026 14:59:15.071395 22622 leader_election.cc:403] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20251026 14:59:15.049055 24320 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:15.011965 24328 consensus_queue.cc:237] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.042352 24322 raft_consensus.cc:2804] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:15.085208 22622 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:15.064064 24406 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:15.040755 24377 consensus_queue.cc:1048] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.042s
I20251026 14:59:15.040697 24410 consensus_queue.cc:1048] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.060s
I20251026 14:59:15.089843 24322 raft_consensus.cc:697] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:15.087373 24386 raft_consensus.cc:2749] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5.
I20251026 14:59:15.143242 24419 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:15.079221 24339 consensus_queue.cc:1048] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20251026 14:59:15.150600 24322 consensus_queue.cc:237] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.151710 24419 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.149806 24327 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:15.144070 24372 raft_consensus.cc:2804] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:15.160943 24372 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:15.161634 24372 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:15.178306 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:15.179482 22727 raft_consensus.cc:2410] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20251026 14:59:15.179965 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:15.196211 24420 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:15.212424 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 127279us
I20251026 14:59:15.214900 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:15.215528 22652 raft_consensus.cc:2468] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 3.
I20251026 14:59:15.216430 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:15.216938 22654 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 3.
I20251026 14:59:15.219090 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:15.219802 22654 raft_consensus.cc:2393] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 248493ec312e4398b828c6cb00e5c4b1 in current term 4: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:15.213824 24419 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:15.203533 24141 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.205696 24420 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.236686 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:15.237092 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:15.239135 22654 raft_consensus.cc:2468] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:15.237565 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:15.240736 22807 raft_consensus.cc:1240] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20251026 14:59:15.241694 22806 raft_consensus.cc:1240] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:15.241786 22657 raft_consensus.cc:2393] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:15.243144 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:15.243290 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:15.244194 22803 raft_consensus.cc:2468] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:15.220712 24411 consensus_queue.cc:1048] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20251026 14:59:15.244628 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:15.245551 22801 raft_consensus.cc:1275] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:15.245327 22622 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:15.247808 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:15.247815 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:15.248503 22655 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:15.251348 24420 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:15.253636 23370 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:15.254567 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:15.255137 23370 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:15.255988 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:15.256269 23370 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:15.257184 24274 raft_consensus.cc:2804] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:15.257633 24274 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:15.258049 24274 raft_consensus.cc:3060] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:15.255285 22658 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:15.243911 22773 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.703s (SENT) [suppressed 70 similar messages]
I20251026 14:59:15.261626 22800 raft_consensus.cc:1275] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:15.262274 22774 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:15.276997 22653 raft_consensus.cc:1240] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:15.234114 24308 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:15.235495 24422 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:15.237867 24339 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:15.237768 24314 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:15.225143 24428 consensus_queue.cc:1048] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
I20251026 14:59:15.294586 22622 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:15.274281 24258 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:15.271466 24427 raft_consensus.cc:2749] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:15.289745 24406 raft_consensus.cc:2804] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:15.274497 24326 raft_consensus.cc:2804] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:15.287508 22727 raft_consensus.cc:1275] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251026 14:59:15.324056 22658 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:15.320355 24419 consensus_queue.cc:1048] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20251026 14:59:15.306372 24153 raft_consensus.cc:2804] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:15.332007 24153 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:15.332562 24153 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:15.307561 24326 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:15.333462 24326 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:15.296672 24314 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.339979 24314 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:15.304708 24406 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:15.291618 24308 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:15.353610 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 63499.6us
I20251026 14:59:15.294457 24339 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.342104 22802 raft_consensus.cc:1240] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251026 14:59:15.304268 24031 consensus_queue.cc:1059] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
W20251026 14:59:15.356201 24431 log.cc:927] Time spent T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.057s user 0.001s sys 0.001s
I20251026 14:59:15.348414 24406 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:15.356756 24423 log.cc:927] Time spent T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.139s user 0.000s sys 0.001s
I20251026 14:59:15.367471 24031 consensus_queue.cc:1243] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer 0d684445287b450a8f738b6ad4b59821 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251026 14:59:15.299767 24422 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:15.368134 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 64064.6us
I20251026 14:59:15.356438 22731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 1139) took 1166 ms. Trace:
I20251026 14:59:15.345458 24427 consensus_queue.cc:1048] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20251026 14:59:15.321486 24315 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.371481 22731 rpcz_store.cc:276] 1026 14:59:14.190276 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:14.190408 (+ 132us) service_pool.cc:225] Handling call
1026 14:59:15.356158 (+1165750us) spinlock_profiling.cc:243] Waited 64.3 ms on lock 0x7b54001e8cd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:15.356350 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":147931904}
I20251026 14:59:15.374877 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:15.360899 24375 log.cc:927] Time spent T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.843s user 0.003s sys 0.000s
I20251026 14:59:15.357590 24108 raft_consensus.cc:3055] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:15.377604 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:15.378540 22650 raft_consensus.cc:1240] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:15.380769 24108 raft_consensus.cc:740] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:15.386005 24308 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:15.385934 22653 raft_consensus.cc:1275] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:15.386534 24108 consensus_queue.cc:260] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.387085 24339 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:15.387589 24108 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:15.387845 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 773911us
I20251026 14:59:15.389832 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:15.391155 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:15.398029 22655 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:15.383625 24141 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:15.350585 24372 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.407152 22773 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:15.407382 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:15.410190 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:15.013929 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:59:15.412369 22655 raft_consensus.cc:1275] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20251026 14:59:15.412665 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:15.413755 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:15.414091 24336 raft_consensus.cc:3060] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:15.414752 24228 raft_consensus.cc:3055] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:15.415140 24228 raft_consensus.cc:740] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:15.409116 24381 raft_consensus.cc:2804] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:15.415928 24356 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:15.416932 24356 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.417891 22807 raft_consensus.cc:1275] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251026 14:59:15.376916 24419 consensus_queue.cc:1059] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.042s
I20251026 14:59:15.398425 24153 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.419438 24237 raft_consensus.cc:3055] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:15.419651 22651 raft_consensus.cc:1240] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251026 14:59:15.419855 24237 raft_consensus.cc:740] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:15.423081 24153 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:15.423559 24356 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:15.424839 24379 consensus_queue.cc:1048] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20251026 14:59:15.416409 24381 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:15.429097 24103 raft_consensus.cc:3055] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:15.429595 24103 raft_consensus.cc:740] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:59:15.400985 24425 log.cc:927] Time spent T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.180s user 0.001s sys 0.000s
I20251026 14:59:15.430627 24103 consensus_queue.cc:260] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.431857 24103 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:15.429651 24079 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:15.399793 24309 consensus_queue.cc:260] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.432458 24079 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:15.420682 24237 consensus_queue.cc:260] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.402745 24422 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:15.437585 24237 raft_consensus.cc:3060] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:15.401628 24315 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:15.441857 24079 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.429446 24381 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:15.415956 24228 consensus_queue.cc:260] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.454434 24228 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:15.455844 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:15.456462 22727 raft_consensus.cc:3060] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:15.447528 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:15.457048 22729 raft_consensus.cc:2376] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5. Re-sending same reply.
I20251026 14:59:15.445734 24418 raft_consensus.cc:3055] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:15.455122 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:15.457635 22729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 1149) took 1264 ms. Trace:
W20251026 14:59:15.457830 24417 log.cc:927] Time spent T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.368s user 0.001s sys 0.010s
I20251026 14:59:15.458039 22731 raft_consensus.cc:2368] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5.
I20251026 14:59:15.458048 22729 rpcz_store.cc:276] 1026 14:59:14.192822 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:14.202824 (+ 10002us) service_pool.cc:225] Handling call
1026 14:59:15.447368 (+1244544us) spinlock_profiling.cc:243] Waited 3.58 ms on lock 0x7b5800250598. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8ce074451 00007fd8ce074302 00007fd8d6f0dad9 00007fd8d6090d30 00007fd8d615984b 00007fd8d61598d1 00007fd8d6107660 00007fd8d6106f44 00007fd8cf6a4834 00007fd8cf6a47c2
1026 14:59:15.457574 (+ 10206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8236928}
I20251026 14:59:15.460186 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:15.460593 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:15.461293 22726 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:15.463201 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:15.446722 24372 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:15.466857 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:15.468580 22725 raft_consensus.cc:2410] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:15.469527 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:15.469993 22728 raft_consensus.cc:1275] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:15.470214 22726 raft_consensus.cc:2368] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5.
I20251026 14:59:15.472040 22726 raft_consensus.cc:1275] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:15.475402 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:15.476810 22728 raft_consensus.cc:1275] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:15.478487 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:15.479089 22725 raft_consensus.cc:1240] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20251026 14:59:15.479207 22732 raft_consensus.cc:2393] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 5: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:15.481629 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:15.482470 22726 raft_consensus.cc:2393] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:15.484691 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:15.485265 22730 raft_consensus.cc:2368] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 5. Current term is 6.
I20251026 14:59:15.487795 22728 raft_consensus.cc:3055] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:15.490486 22732 raft_consensus.cc:1275] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:15.488210 22728 raft_consensus.cc:740] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:15.457645 24418 raft_consensus.cc:740] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:15.497314 22725 raft_consensus.cc:1240] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
W20251026 14:59:15.503422 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:15.492198 22728 consensus_queue.cc:260] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.512548 22728 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:15.524847 24258 raft_consensus.cc:2749] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:15.525441 24294 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:15.527087 24079 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:15.528205 22731 raft_consensus.cc:2410] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251026 14:59:15.528983 24418 consensus_queue.cc:260] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.532603 24379 consensus_queue.cc:1048] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251026 14:59:15.534756 22804 raft_consensus.cc:1275] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 2. (index mismatch)
I20251026 14:59:15.536718 22727 raft_consensus.cc:2468] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:15.536758 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:15.537655 22808 raft_consensus.cc:2468] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:59:15.538157 22727 raft_consensus.cc:1275] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 2. (term mismatch)
I20251026 14:59:15.538535 22727 pending_rounds.cc:85] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e: Aborting all ops after (but not including) 1
I20251026 14:59:15.538748 22727 pending_rounds.cc:107] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20251026 14:59:15.536886 22804 raft_consensus.cc:1240] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:15.539017 22727 raft_consensus.cc:2889] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:15.539348 22727 pending_rounds.cc:107] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20251026 14:59:15.539587 22727 raft_consensus.cc:2889] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:15.539851 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:15.541043 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:15.541944 22804 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:15.543315 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:15.477612 24326 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:15.559645 24326 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:15.560473 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:15.551064 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:15.553608 22773 leader_election.cc:336] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.810s (SENT)
W20251026 14:59:15.562831 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:15.569254 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:15.569983 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:15.570809 22657 raft_consensus.cc:2468] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:59:15.571190 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:15.570022 22808 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:15.579830 22657 raft_consensus.cc:1275] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:15.580669 22697 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:15.583040 22774 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:15.584486 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:15.585572 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:15.585915 22804 raft_consensus.cc:2393] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:15.588238 22697 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:15.586215 22800 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:15.597509 22650 raft_consensus.cc:1275] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20251026 14:59:15.597944 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:15.598657 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:15.602579 22657 raft_consensus.cc:1275] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20251026 14:59:15.603121 22657 pending_rounds.cc:85] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29: Aborting all ops after (but not including) 2
I20251026 14:59:15.603392 22657 pending_rounds.cc:107] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20251026 14:59:15.603876 22657 raft_consensus.cc:2889] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:15.498790 24274 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:15.608439 22774 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:15.609750 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:15.616715 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:15.619788 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:15.621097 22804 raft_consensus.cc:2393] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:15.624068 22697 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:15.628103 22808 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:15.628177 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:15.629526 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:15.600911 24328 raft_consensus.cc:2804] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:15.600167 24237 raft_consensus.cc:2804] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:59:15.634233 22650 raft_consensus.cc:1275] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20251026 14:59:15.634644 22650 pending_rounds.cc:85] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29: Aborting all ops after (but not including) 2
I20251026 14:59:15.634865 22650 pending_rounds.cc:107] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251026 14:59:15.635088 22650 raft_consensus.cc:2889] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:15.635273 22808 raft_consensus.cc:1240] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20251026 14:59:15.623979 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:15.637555 22804 raft_consensus.cc:2393] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 4: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:15.613631 24142 raft_consensus.cc:2804] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:15.610551 24322 consensus_queue.cc:1059] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20251026 14:59:15.608107 24294 consensus_queue.cc:1059] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20251026 14:59:15.608028 24115 consensus_queue.cc:1048] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.039s
I20251026 14:59:15.633328 24397 raft_consensus.cc:2804] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:15.630577 24341 raft_consensus.cc:2804] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:15.634582 24225 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:15.637863 24237 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:15.652633 24142 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:15.700603 23402 raft_consensus.cc:1275] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 4. (term mismatch)
I20251026 14:59:15.654451 23400 raft_consensus.cc:1240] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:15.700723 24142 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:15.651062 23399 raft_consensus.cc:1240] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:15.653517 24328 raft_consensus.cc:697] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:15.681521 24237 raft_consensus.cc:3060] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:59:15.704586 24406 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.701629 23402 pending_rounds.cc:85] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1: Aborting all ops after (but not including) 3
W20251026 14:59:15.749500 22697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:15.712077 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:15.703853 22774 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:15.701330 24224 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:15.683718 24341 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:15.762418 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 58659.1us
I20251026 14:59:15.759909 22806 raft_consensus.cc:1275] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20251026 14:59:15.769644 24450 log.cc:927] Time spent T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.118s user 0.000s sys 0.001s
W20251026 14:59:15.761953 22604 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 9339edd1ecbf42e3a5e953b2b47f52f2, TS 562bb1e1c8e445b8baecd29c3669658e -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:15.771278 22604 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 9339edd1ecbf42e3a5e953b2b47f52f2, TS 562bb1e1c8e445b8baecd29c3669658e -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:15.764468 24459 log.cc:927] Time spent T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.059s user 0.001s sys 0.001s
I20251026 14:59:15.701586 24225 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.775053 24225 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:15.764670 23402 pending_rounds.cc:107] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 3.4
I20251026 14:59:15.776716 24420 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:15.777318 22799 raft_consensus.cc:1240] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:15.777298 24420 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.779263 22622 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:15.780922 24325 raft_consensus.cc:2804] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:15.759375 24336 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:15.781518 24325 raft_consensus.cc:697] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:15.781679 24336 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.782406 24325 consensus_queue.cc:237] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.783707 24079 consensus_queue.cc:1048] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:59:15.786482 24313 raft_consensus.cc:3055] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:15.786924 24313 raft_consensus.cc:740] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
W20251026 14:59:15.787524 24448 log.cc:927] Time spent T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20251026 14:59:15.787845 22799 raft_consensus.cc:1275] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:15.706677 24397 raft_consensus.cc:697] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:15.774957 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:15.792836 22734 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:15.817934 22800 raft_consensus.cc:2393] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:15.817986 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:15.791983 24224 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.821103 22799 raft_consensus.cc:1275] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:15.821581 24224 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:15.787788 24313 consensus_queue.cc:260] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.707212 24103 consensus_queue.cc:1048] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.082s
I20251026 14:59:15.823215 24313 raft_consensus.cc:3060] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:15.824532 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:15.825155 22700 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:15.825656 24352 consensus_queue.cc:1059] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:59:15.828588 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:15.836516 22726 raft_consensus.cc:2468] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:15.834517 22808 raft_consensus.cc:1275] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20251026 14:59:15.837239 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:15.759810 24322 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:15.839828 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:15.840271 22650 raft_consensus.cc:1275] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20251026 14:59:15.840646 22650 pending_rounds.cc:85] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29: Aborting all ops after (but not including) 2
I20251026 14:59:15.840867 22650 pending_rounds.cc:107] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251026 14:59:15.841086 22650 raft_consensus.cc:2889] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251026 14:59:15.816049 24409 log.cc:927] Time spent T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20251026 14:59:15.776721 23402 raft_consensus.cc:2917] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Aborting config change with OpId 3.4: Aborted: Op aborted by new leader
I20251026 14:59:15.841260 22726 raft_consensus.cc:2410] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20251026 14:59:15.842854 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 74645.2us
I20251026 14:59:15.790514 24341 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:15.845372 22728 raft_consensus.cc:1275] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20251026 14:59:15.846084 22700 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:15.783738 24336 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:15.856137 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:15.856643 22734 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:15.857219 23402 raft_consensus.cc:2889] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:15.826304 24397 consensus_queue.cc:237] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.835598 24440 consensus_queue.cc:1048] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20251026 14:59:15.877038 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:15.910799 22734 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:15.849344 22728 pending_rounds.cc:85] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e: Aborting all ops after (but not including) 2
I20251026 14:59:15.877358 22700 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:15.853816 24224 raft_consensus.cc:2749] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:15.913851 22728 pending_rounds.cc:107] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20251026 14:59:15.914211 22728 raft_consensus.cc:2889] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:15.915655 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:15.924418 22734 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:15.926625 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:15.960942 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:15.961292 22700 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:15.961921 22728 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:15.965828 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:15.847522 24322 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.969267 24294 consensus_queue.cc:1048] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20251026 14:59:16.006296 24333 consensus_queue.cc:1059] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251026 14:59:16.008889 24333 raft_consensus.cc:3055] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:15.979013 22700 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:15.879902 24153 consensus_queue.cc:1048] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.090s
I20251026 14:59:15.870959 24328 consensus_queue.cc:237] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:15.876859 24225 raft_consensus.cc:2749] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:15.859434 24228 consensus_queue.cc:1048] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.107s
I20251026 14:59:15.858206 24315 consensus_queue.cc:1048] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.104s
I20251026 14:59:15.881140 24274 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:15.907377 24455 raft_consensus.cc:2804] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:15.992486 22800 raft_consensus.cc:1275] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:15.986004 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:15.989779 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:16.002528 22801 raft_consensus.cc:1240] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:16.016994 22700 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:15.990592 22725 raft_consensus.cc:1275] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20251026 14:59:15.875037 24416 consensus_queue.cc:1048] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.155s
W20251026 14:59:16.053256 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:16.055087 22725 pending_rounds.cc:85] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e: Aborting all ops after (but not including) 2
I20251026 14:59:16.031862 22734 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:16.055405 22725 pending_rounds.cc:107] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20251026 14:59:16.055770 22804 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
W20251026 14:59:15.981775 24451 log.cc:927] Time spent T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.188s user 0.001s sys 0.000s
I20251026 14:59:16.058789 22700 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:16.015484 24327 raft_consensus.cc:2749] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20251026 14:59:15.971163 24457 log.cc:927] Time spent T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
I20251026 14:59:15.994133 24420 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:15.973232 24447 log.cc:927] Time spent T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.224s user 0.000s sys 0.001s
I20251026 14:59:15.977509 24406 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:16.059540 22803 raft_consensus.cc:1275] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251026 14:59:16.107609 22807 raft_consensus.cc:1240] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:16.108947 22807 raft_consensus.cc:1240] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:15.987604 24309 raft_consensus.cc:3060] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:16.110296 24439 log.cc:927] Time spent T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.582s user 0.002s sys 0.000s
I20251026 14:59:16.060552 22725 raft_consensus.cc:2889] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:16.060348 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:16.113787 22731 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:16.114879 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:16.115599 22731 raft_consensus.cc:2376] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5. Re-sending same reply.
I20251026 14:59:16.116601 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:16.064940 22700 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:15.992278 24381 consensus_meta.cc:287] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.538s user 0.008s sys 0.000s
I20251026 14:59:16.000494 24322 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:15.997655 24224 raft_consensus.cc:2749] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:16.067943 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:16.123703 22734 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:16.072482 24356 raft_consensus.cc:2749] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:59:16.125248 22805 raft_consensus.cc:1240] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:16.070286 22800 raft_consensus.cc:1275] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20251026 14:59:16.130566 22800 pending_rounds.cc:85] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821: Aborting all ops after (but not including) 2
I20251026 14:59:16.130920 22800 pending_rounds.cc:107] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20251026 14:59:16.131210 22800 raft_consensus.cc:2889] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:16.010527 24333 raft_consensus.cc:740] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:16.071249 22622 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:16.136384 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 65226.4us
W20251026 14:59:16.138330 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:16.067515 24422 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:15.969878 24352 raft_consensus.cc:2749] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:16.059075 24225 raft_consensus.cc:2749] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:16.071740 24032 consensus_queue.cc:1048] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20251026 14:59:16.081856 24341 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.089814 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:16.159932 22730 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:16.102370 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:16.035861 22728 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:16.162181 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:16.163007 22728 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:16.075291 24308 consensus_queue.cc:1048] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20251026 14:59:16.103804 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:16.166177 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:16.105572 24444 log.cc:927] Time spent T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.580s user 0.003s sys 0.001s
W20251026 14:59:16.120645 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 55797.7us
I20251026 14:59:16.143487 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:16.169963 22731 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:16.149516 22734 raft_consensus.cc:2468] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:16.156128 22803 raft_consensus.cc:1275] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:16.061390 24463 consensus_queue.cc:1048] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.096s
I20251026 14:59:16.124411 24327 raft_consensus.cc:2749] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:16.171792 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:16.172691 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:16.123978 24381 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.174562 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:16.174682 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:16.156692 24356 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:16.172118 22700 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:16.219170 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:16.187451 22726 raft_consensus.cc:2393] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 5: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:16.153647 24325 raft_consensus.cc:2804] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:16.215096 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:16.169775 24225 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:16.221108 22728 raft_consensus.cc:2393] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 5: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:16.215302 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:16.213878 22727 raft_consensus.cc:2468] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:16.221990 22734 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:16.160218 24416 consensus_queue.cc:1059] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20251026 14:59:16.154877 24385 consensus_queue.cc:1048] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20251026 14:59:16.172152 24440 consensus_queue.cc:1048] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20251026 14:59:16.252697 22804 raft_consensus.cc:1240] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:16.235405 22730 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:16.237752 22731 raft_consensus.cc:1275] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:16.254165 22731 raft_consensus.cc:3055] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:16.254491 22731 raft_consensus.cc:740] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:16.232867 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:16.255992 22650 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:16.257395 24408 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:16.257880 24408 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.235814 24225 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.245491 22727 raft_consensus.cc:1275] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:16.246141 24312 consensus_queue.cc:1048] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:59:16.246424 24327 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:16.174052 24311 consensus_queue.cc:1059] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.074s
I20251026 14:59:16.173705 24333 consensus_queue.cc:260] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:16.224388 24325 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:16.282127 24225 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:16.214565 24420 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:16.282860 24476 raft_consensus.cc:2749] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:16.282961 24420 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.155879 24422 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.285802 24381 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:16.286732 24422 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:16.214358 24397 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:16.265661 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:16.288880 22657 raft_consensus.cc:3060] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
W20251026 14:59:16.270195 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:16.273723 24341 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:16.260303 24346 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:16.235423 24356 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.183384 24352 consensus_queue.cc:1059] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20251026 14:59:16.171172 24142 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.255147 22731 consensus_queue.cc:260] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.298283 22731 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:16.282243 24325 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:16.262636 22727 raft_consensus.cc:1275] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:16.227212 24237 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.301463 22727 raft_consensus.cc:1240] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:16.259860 24408 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:16.286778 24311 raft_consensus.cc:3055] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
W20251026 14:59:16.302881 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 46351.8us
I20251026 14:59:16.309504 22734 raft_consensus.cc:1240] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:16.303321 24031 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:16.314071 22727 raft_consensus.cc:1240] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
W20251026 14:59:16.312603 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 140539us
I20251026 14:59:16.297551 24420 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:16.313802 22728 raft_consensus.cc:1240] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
I20251026 14:59:16.319590 22731 raft_consensus.cc:1275] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20251026 14:59:16.319952 22731 pending_rounds.cc:85] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e: Aborting all ops after (but not including) 2
I20251026 14:59:16.320770 22731 pending_rounds.cc:107] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251026 14:59:16.321627 22731 raft_consensus.cc:2889] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:16.317436 22700 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:16.307632 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:16.320698 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:16.327644 24142 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:16.287534 24397 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.306493 24311 raft_consensus.cc:740] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:16.329840 24397 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:16.330226 24311 consensus_queue.cc:260] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:16.313335 24385 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:16.331481 24385 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.332417 22726 raft_consensus.cc:1275] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:16.331719 24311 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:16.333784 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:16.281173 24327 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.313598 24341 raft_consensus.cc:3055] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:16.298190 24346 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.338881 24479 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:16.339643 24356 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:16.339430 24479 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:16.339888 22726 raft_consensus.cc:1275] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 4. (term mismatch)
I20251026 14:59:16.340374 22726 pending_rounds.cc:85] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e: Aborting all ops after (but not including) 3
I20251026 14:59:16.340653 22726 pending_rounds.cc:107] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 3.4
I20251026 14:59:16.340932 22726 raft_consensus.cc:2917] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Aborting config change with OpId 3.4: Aborted: Op aborted by new leader
I20251026 14:59:16.343078 24327 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:16.343816 22726 raft_consensus.cc:2889] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:16.323119 24031 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.346469 24031 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:16.337428 24341 raft_consensus.cc:740] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:16.347697 24341 consensus_queue.cc:260] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.348896 24341 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:16.351590 24142 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:16.354058 24142 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.355093 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:16.355882 24142 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:16.351869 24479 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:16.352270 24478 raft_consensus.cc:2749] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:16.360976 22733 raft_consensus.cc:1240] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
W20251026 14:59:16.365490 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 48196.1us
I20251026 14:59:16.367237 22624 leader_election.cc:400] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507) with higher term. Message: Invalid argument: T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5.
I20251026 14:59:16.368815 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:16.371430 22653 raft_consensus.cc:1275] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 14:59:16.374492 22650 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 4.
I20251026 14:59:16.351306 22730 raft_consensus.cc:1275] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:16.376601 22624 leader_election.cc:400] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Vote denied by peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507) with higher term. Message: Invalid argument: T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5.
I20251026 14:59:16.318017 22657 raft_consensus.cc:2468] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 4.
I20251026 14:59:16.378110 22656 raft_consensus.cc:1240] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:16.378875 22731 raft_consensus.cc:1275] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251026 14:59:16.380627 22731 raft_consensus.cc:1275] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20251026 14:59:16.443567 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 80217us
I20251026 14:59:16.445715 22650 raft_consensus.cc:1240] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:16.445770 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:16.447645 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:16.452422 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:16.457196 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:59:16.460588 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 111389us
W20251026 14:59:16.461122 22773 leader_election.cc:336] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.790s (SENT)
I20251026 14:59:16.461443 22773 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:16.463845 22773 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:16.466580 22624 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:16.467181 22658 raft_consensus.cc:1275] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251026 14:59:16.467259 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:16.469802 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:16.370848 24337 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:16.486549 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:16.493341 22726 raft_consensus.cc:1275] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:16.478698 22624 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:16.482410 22700 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:16.496771 22624 leader_election.cc:400] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Vote denied by peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507) with higher term. Message: Invalid argument: T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 5. Current term is 6.
I20251026 14:59:16.458173 24385 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:16.470299 24420 raft_consensus.cc:2749] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:16.466465 24397 raft_consensus.cc:2749] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20251026 14:59:16.513338 22774 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.656s user 0.036s sys 0.032s
W20251026 14:59:16.514916 22774 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.660s user 0.036s sys 0.032s
W20251026 14:59:16.515275 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 660303us
I20251026 14:59:16.513790 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:16.523190 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 59443us
I20251026 14:59:16.484177 24332 raft_consensus.cc:2749] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:16.527863 22773 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:16.530054 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:16.465945 24336 consensus_queue.cc:1048] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.042s
I20251026 14:59:16.489789 24479 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:16.469476 24440 consensus_queue.cc:1048] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.047s
I20251026 14:59:16.542428 24333 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:16.490078 24142 raft_consensus.cc:493] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:16.554972 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:16.549692 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 769) took 1057 ms. Trace:
I20251026 14:59:16.556257 22725 rpcz_store.cc:276] 1026 14:59:15.492069 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:15.511435 (+ 19366us) service_pool.cc:225] Handling call
1026 14:59:15.990229 (+478794us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:16.112549 (+122320us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:16.549476 (+436927us) spinlock_profiling.cc:243] Waited 470 ms on lock 0x7b540040e2d0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:16.549633 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1081842048}
I20251026 14:59:16.557525 22658 raft_consensus.cc:1240] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20251026 14:59:16.557801 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 65331.8us
I20251026 14:59:16.563100 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:16.496673 24337 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.567987 22773 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:16.518230 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:16.569545 22773 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:16.484992 24408 consensus_queue.cc:1048] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.046s
I20251026 14:59:16.581596 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:16.582930 22657 raft_consensus.cc:2468] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:16.582332 22725 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:16.493168 24478 consensus_queue.cc:1059] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.073s
I20251026 14:59:16.590517 22773 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:16.591758 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:16.593081 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:16.594278 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:16.595065 22725 raft_consensus.cc:2393] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:16.537984 24406 consensus_queue.cc:1048] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
I20251026 14:59:16.538290 24352 consensus_queue.cc:1048] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20251026 14:59:16.599426 24478 raft_consensus.cc:3055] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:16.599792 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:16.573760 24420 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:16.601176 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:16.601646 24420 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.602612 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:16.603909 24420 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:16.603950 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:16.599915 24478 raft_consensus.cc:740] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:16.606004 24478 consensus_queue.cc:260] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:16.565871 24332 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:16.607645 24332 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.608039 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:16.609570 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:16.609720 24332 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:16.568965 24142 raft_consensus.cc:515] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:16.610893 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:16.612195 24142 leader_election.cc:290] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:16.612097 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:16.615002 24377 raft_consensus.cc:2804] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:16.615672 24377 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:16.616168 24377 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:16.568413 24427 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:16.617190 24427 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:59:16.606701 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:16.619716 22773 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:16.634907 22773 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:16.635147 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:16.636117 22803 raft_consensus.cc:2468] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:59:16.636348 22773 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:16.538378 24385 consensus_queue.cc:1048] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.058s
I20251026 14:59:16.644562 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:16.640463 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:16.573570 24315 raft_consensus.cc:2749] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251026 14:59:16.529810 24422 consensus_queue.cc:1048] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20251026 14:59:16.645440 22725 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:16.565294 24309 consensus_queue.cc:1048] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.073s
I20251026 14:59:16.607656 24478 raft_consensus.cc:3060] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:16.660890 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:16.663455 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:16.664323 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 148744us
I20251026 14:59:16.580428 24479 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.585647 24308 consensus_queue.cc:1048] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.076s
I20251026 14:59:16.643998 24420 raft_consensus.cc:2749] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:16.639497 24346 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:16.672869 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 119423us
I20251026 14:59:16.683933 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:16.725988 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 51772.7us
I20251026 14:59:16.668335 24328 raft_consensus.cc:2749] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:16.652730 24325 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.733789 22624 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:16.728099 24352 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:16.730603 24476 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:16.732388 22697 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:16.734978 22624 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:16.733850 22727 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:16.745828 22733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 772) took 1192 ms. Trace:
I20251026 14:59:16.755326 22733 rpcz_store.cc:276] 1026 14:59:15.553731 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:16.226011 (+672280us) service_pool.cc:225] Handling call
1026 14:59:16.360733 (+134722us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:16.363000 (+ 2267us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:16.745657 (+382657us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b54002db750. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:16.745770 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":247001472}
I20251026 14:59:16.746555 22806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 976) took 1171 ms. Trace:
W20251026 14:59:16.759307 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:16.758018 22806 rpcz_store.cc:276] 1026 14:59:15.575610 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:15.774710 (+199100us) service_pool.cc:225] Handling call
1026 14:59:15.774981 (+ 271us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:16.001516 (+226535us) raft_consensus.cc:1534] Early marking committed up to index 3
1026 14:59:16.001826 (+ 310us) log.cc:844] Serialized 14 byte log entry
1026 14:59:16.114387 (+112561us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:16.114467 (+ 80us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:16.114526 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:16.746360 (+631834us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b54001be754. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:16.746401 (+ 41us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:16.746503 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":548,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":4624,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":14937,"spinlock_wait_cycles":520433920,"thread_start_us":112175,"threads_started":1,"wal-append.queue_time_us":169425}
I20251026 14:59:16.746568 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 776) took 1191 ms. Trace:
I20251026 14:59:16.760479 22734 rpcz_store.cc:276] 1026 14:59:15.555073 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:16.229619 (+674546us) service_pool.cc:225] Handling call
1026 14:59:16.230045 (+ 426us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:16.312073 (+ 82028us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:16.746360 (+434287us) spinlock_profiling.cc:243] Waited 79.2 ms on lock 0x7b54000b7cd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:16.746519 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":182194944}
W20251026 14:59:16.748200 22700 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:16.765556 22700 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:16.751338 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:16.752535 22697 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:16.769622 22700 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:16.735278 24483 log.cc:927] Time spent T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.146s user 0.000s sys 0.001s
I20251026 14:59:16.771872 22734 raft_consensus.cc:1275] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:16.773423 22731 raft_consensus.cc:1275] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251026 14:59:16.752472 24294 raft_consensus.cc:2749] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251026 14:59:16.734619 24482 log.cc:927] Time spent T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20251026 14:59:16.779783 22624 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:16.782542 22651 raft_consensus.cc:2368] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6.
I20251026 14:59:16.783794 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:16.737437 24403 raft_consensus.cc:2749] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:16.788610 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:16.736469 24104 raft_consensus.cc:2749] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:16.770282 24463 raft_consensus.cc:2804] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:16.749758 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:16.731367 24332 consensus_queue.cc:1048] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20251026 14:59:16.737567 24466 raft_consensus.cc:2749] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:16.801824 22727 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:16.811357 22808 raft_consensus.cc:1275] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251026 14:59:16.812951 22808 raft_consensus.cc:1240] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:16.785223 24031 raft_consensus.cc:2804] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:16.738761 24352 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.794431 24427 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:16.741905 24476 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.819419 22624 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:16.792584 24350 consensus_queue.cc:1048] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20251026 14:59:16.818835 24494 raft_consensus.cc:2749] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251026 14:59:16.828336 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 192063us
I20251026 14:59:16.816421 24337 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:16.818271 24104 raft_consensus.cc:2804] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:16.820470 24294 raft_consensus.cc:2804] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:16.818210 24031 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:16.840226 22773 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:16.840395 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 766) took 1360 ms. Trace:
I20251026 14:59:16.851882 22732 rpcz_store.cc:276] 1026 14:59:15.473611 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:15.491214 (+ 17603us) service_pool.cc:225] Handling call
1026 14:59:15.503142 (+ 11928us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:15.503387 (+ 245us) raft_consensus.cc:1534] Early marking committed up to index 3
1026 14:59:15.612133 (+108746us) log.cc:844] Serialized 14 byte log entry
1026 14:59:16.747248 (+1135115us) spinlock_profiling.cc:243] Waited 4.31 ms on lock 0x7b1800053b60. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8d0d0655c 00007fd8ce09497b 00007fd8d6e6d219 00007fd8ce0b0c4b 00007fd8ce0aeeef 00007fd8ce0b11b2 00007fd8d0cf82bb 00007fd8d0cfddc3 00007fd8d0cfe2da
1026 14:59:16.833623 (+ 86375us) log.cc:844] Serialized 14 byte log entry
1026 14:59:16.833907 (+ 284us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:16.833969 (+ 62us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:16.834026 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:16.834111 (+ 85us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:16.834253 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5507,"prepare.queue_time_us":146102,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":59,"raft.queue_time_us":2793,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":6329,"spinlock_wait_cycles":9914624,"thread_start_us":232744,"threads_started":3,"wal-append.queue_time_us":258057}
I20251026 14:59:16.828109 24397 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:16.863104 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:16.866431 22657 raft_consensus.cc:2393] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 5: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:16.863839 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:16.866508 22730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 783) took 1292 ms. Trace:
I20251026 14:59:16.867018 22730 rpcz_store.cc:276] 1026 14:59:15.586596 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:16.264675 (+678079us) service_pool.cc:225] Handling call
1026 14:59:16.265016 (+ 341us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:16.375490 (+110474us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:16.866274 (+490784us) spinlock_profiling.cc:243] Waited 85.4 ms on lock 0x7b540030b1d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:16.866447 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":196331776}
I20251026 14:59:16.869671 22656 raft_consensus.cc:1275] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251026 14:59:16.869787 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:16.832508 22802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 881) took 1600 ms. Trace:
I20251026 14:59:16.870675 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:16.838305 24104 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:16.839460 24385 raft_consensus.cc:2749] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251026 14:59:16.874334 22656 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:16.842832 24294 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:16.876827 22624 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:16.872442 24466 raft_consensus.cc:1064] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821: attempting to promote NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 to VOTER
I20251026 14:59:16.887441 22800 raft_consensus.cc:1240] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5. Ops: []
W20251026 14:59:16.889497 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:16.889739 22730 raft_consensus.cc:1275] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20251026 14:59:16.890525 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:16.875819 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 123367us
I20251026 14:59:16.841956 24422 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:16.861653 24031 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:16.851073 24350 consensus_queue.cc:1048] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.034s
I20251026 14:59:16.897410 22732 raft_consensus.cc:1275] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251026 14:59:16.904240 22697 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:16.909983 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:16.878612 24104 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:16.850749 24346 consensus_queue.cc:1048] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20251026 14:59:16.860924 24403 consensus_queue.cc:1048] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
I20251026 14:59:16.891417 24490 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:16.913704 22800 raft_consensus.cc:1275] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251026 14:59:16.891288 24384 raft_consensus.cc:2749] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:16.904845 24385 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:16.921131 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:16.892621 24294 raft_consensus.cc:3060] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
W20251026 14:59:16.922040 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 152517us
W20251026 14:59:16.906117 24466 raft_consensus.cc:1068] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821: Unable to promote non-voter 248493ec312e4398b828c6cb00e5c4b1: Illegal state: Leader has not yet committed an operation in its own term
I20251026 14:59:16.917044 22697 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:16.939851 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:16.914595 24308 raft_consensus.cc:2749] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:16.945744 22624 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
W20251026 14:59:16.932367 24341 consensus_meta.cc:287] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.583s user 0.014s sys 0.006s
I20251026 14:59:16.943109 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:16.943773 22733 raft_consensus.cc:1275] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251026 14:59:16.881738 24427 consensus_queue.cc:1048] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20251026 14:59:16.943945 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:16.916254 24479 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:16.950855 22732 raft_consensus.cc:1275] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:16.917347 24325 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:16.891644 24495 consensus_queue.cc:1048] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20251026 14:59:16.954581 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:16.942879 22697 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:16.961730 22733 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:16.958514 22725 raft_consensus.cc:2410] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251026 14:59:16.910333 24422 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.960745 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:16.873678 24397 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.964098 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:16.974629 22656 raft_consensus.cc:2468] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:16.873584 22802 rpcz_store.cc:276] 1026 14:59:15.227462 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:15.227888 (+ 426us) service_pool.cc:225] Handling call
1026 14:59:15.237549 (+ 9661us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:15.367102 (+129553us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:16.816247 (+1449145us) spinlock_profiling.cc:243] Waited 60.3 ms on lock 0x7b540020b7d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:16.827352 (+ 11105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":138749824}
I20251026 14:59:16.945623 24406 raft_consensus.cc:2804] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:16.971557 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:16.977095 22697 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:16.977174 22732 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:16.916865 23370 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:16.978538 22697 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:16.955256 24353 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:16.956846 24333 raft_consensus.cc:2749] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:16.985343 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:16.986264 22732 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:16.990346 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:16.989152 24479 raft_consensus.cc:2749] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:16.993250 24506 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:16.993902 24506 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:16.943419 24385 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.000293 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:17.005228 22733 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:17.010584 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:17.014138 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 63257.9us
I20251026 14:59:16.959297 24332 consensus_queue.cc:1059] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20251026 14:59:16.959154 24384 consensus_queue.cc:1059] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.033s
I20251026 14:59:17.008473 24455 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
W20251026 14:59:17.019656 24500 log.cc:927] Time spent T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20251026 14:59:17.019518 24346 consensus_queue.cc:1048] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:59:17.020830 24385 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:17.022580 24325 consensus_queue.cc:1048] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:59:17.023531 24350 consensus_queue.cc:1048] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:59:17.024734 22624 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:17.026393 24501 log.cc:927] Time spent T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.124s user 0.001s sys 0.000s
I20251026 14:59:17.057384 22624 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:17.027233 24495 consensus_queue.cc:1048] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:17.029948 22622 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:17.031026 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:17.032603 22659 raft_consensus.cc:1275] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20251026 14:59:17.034936 24498 log.cc:927] Time spent T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20251026 14:59:16.989861 24308 raft_consensus.cc:2749] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:16.987534 24353 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:16.974957 24352 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:16.998265 24341 raft_consensus.cc:2749] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251026 14:59:17.051097 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:16.932955 24490 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.008312 24506 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:17.045497 24491 raft_consensus.cc:2804] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:17.048626 24455 raft_consensus.cc:3060] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:17.053696 24403 raft_consensus.cc:3055] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:17.051692 24476 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:17.050098 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 141700us
W20251026 14:59:17.079694 24441 log.cc:927] Time spent T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 1.514s user 0.004s sys 0.000s
I20251026 14:59:17.082701 24515 raft_consensus.cc:2804] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:17.081038 24341 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:17.066063 24115 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:17.089421 22653 raft_consensus.cc:2410] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 6 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:17.090344 23370 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:17.091153 22552 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38880 (request call id 172) took 4599 ms. Trace:
I20251026 14:59:17.070015 24353 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:17.075490 24332 raft_consensus.cc:2804] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:17.109910 24345 raft_consensus.cc:3055] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:17.082267 24403 raft_consensus.cc:740] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:17.084200 24491 raft_consensus.cc:697] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 LEADER]: Becoming Leader. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:17.116686 24490 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:17.117583 24513 log.cc:927] Time spent T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20251026 14:59:17.103987 22552 rpcz_store.cc:276] 1026 14:59:12.491782 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:12.492000 (+ 218us) service_pool.cc:225] Handling call
1026 14:59:17.091088 (+4599088us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 14:59:13.395120 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 14:59:13.469781 (+ 74661us) write_op.cc:432] Acquiring schema lock in shared mode
1026 14:59:13.469835 (+ 54us) write_op.cc:435] Acquired schema lock
1026 14:59:13.469870 (+ 35us) tablet.cc:660] Decoding operations
1026 14:59:13.471342 (+ 1472us) write_op.cc:620] Acquiring the partition lock for write op
1026 14:59:13.471426 (+ 84us) write_op.cc:641] Partition lock acquired for write op
1026 14:59:13.471471 (+ 45us) tablet.cc:683] Acquiring locks for 11 operations
1026 14:59:13.472830 (+ 1359us) tablet.cc:699] Row locks acquired
1026 14:59:13.472862 (+ 32us) write_op.cc:260] PREPARE: finished
1026 14:59:13.473065 (+ 203us) write_op.cc:270] Start()
1026 14:59:13.473237 (+ 172us) write_op.cc:276] Timestamp: P: 1761490753472997 usec, L: 0
1026 14:59:13.505421 (+ 32184us) op_driver.cc:348] REPLICATION: starting
1026 14:59:13.556841 (+ 51420us) log.cc:844] Serialized 4616 byte log entry
1026 14:59:13.674815 (+117974us) op_driver.cc:464] REPLICATION: finished
1026 14:59:13.837761 (+162946us) write_op.cc:301] APPLY: starting
1026 14:59:13.842082 (+ 4321us) tablet.cc:1365] starting BulkCheckPresence
1026 14:59:13.880081 (+ 37999us) tablet.cc:1368] finished BulkCheckPresence
1026 14:59:13.880226 (+ 145us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 14:59:13.881691 (+ 1465us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 14:59:13.881949 (+ 258us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
1026 14:59:13.890974 (+ 9025us) write_op.cc:312] APPLY: finished
1026 14:59:13.960296 (+ 69322us) log.cc:844] Serialized 82 byte log entry
1026 14:59:13.960476 (+ 180us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:13.960603 (+ 127us) write_op.cc:454] Released schema lock
1026 14:59:13.960651 (+ 48us) write_op.cc:341] FINISH: Updating metrics
1026 14:59:13.960960 (+ 309us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:13.960985 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":22207,"reported_tablets":27,"child_traces":[["op",{"apply.queue_time_us":140708,"apply.run_cpu_time_us":4068,"apply.run_wall_time_us":134877,"num_ops":11,"prepare.queue_time_us":182512,"prepare.run_cpu_time_us":24809,"prepare.run_wall_time_us":226858,"raft.queue_time_us":50767,"raft.run_cpu_time_us":36219,"raft.run_wall_time_us":105393,"replication_time_us":169169,"thread_start_us":250615,"threads_started":4,"wal-append.queue_time_us":64291,"wal-append.run_cpu_time_us":15464,"wal-append.run_wall_time_us":1349718}]]}
W20251026 14:59:17.125944 24496 log.cc:927] Time spent T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.217s user 0.002s sys 0.000s
I20251026 14:59:17.111419 24332 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:17.130285 24332 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:17.132298 24104 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.133631 22650 raft_consensus.cc:1240] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20251026 14:59:17.137662 22700 leader_election.cc:400] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6.
I20251026 14:59:17.138056 22700 leader_election.cc:403] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
W20251026 14:59:17.141402 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:17.111997 24345 raft_consensus.cc:740] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:17.102960 24515 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:17.106895 24115 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.145892 24479 raft_consensus.cc:2749] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6.
W20251026 14:59:17.155831 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 98588.8us
I20251026 14:59:17.104965 24341 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.134756 24325 raft_consensus.cc:2749] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:17.157621 22624 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:17.142359 24481 raft_consensus.cc:2804] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:17.143764 24490 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:17.146119 24495 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:17.162856 22697 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
W20251026 14:59:17.145015 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 54795.3us
W20251026 14:59:17.169219 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 102442us
I20251026 14:59:17.155795 24422 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:17.174364 22697 leader_election.cc:304] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:17.155169 24515 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:17.172071 24311 consensus_meta.cc:287] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.811s user 0.006s sys 0.007s
I20251026 14:59:17.182250 24478 raft_consensus.cc:2749] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:17.198041 24325 consensus_queue.cc:1048] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:17.199102 24325 consensus_queue.cc:1243] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer 0d684445287b450a8f738b6ad4b59821 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251026 14:59:17.200603 24325 consensus_queue.cc:1059] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:59:17.181592 24352 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:17.204367 24352 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.179143 24479 raft_consensus.cc:2749] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:17.175706 24481 raft_consensus.cc:697] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 LEADER]: Becoming Leader. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:17.180177 24519 raft_consensus.cc:2804] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:17.187445 24491 consensus_queue.cc:237] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:17.221204 24461 log.cc:927] Time spent T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 1.204s user 0.000s sys 0.006s
I20251026 14:59:17.170665 24495 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.221801 24104 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:17.169113 24490 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:17.216141 24377 consensus_meta.cc:287] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.599s user 0.004s sys 0.009s
I20251026 14:59:17.229605 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:17.236531 22733 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:17.237630 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:17.239707 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:17.240590 22733 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:17.235155 22731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1187) took 1410 ms. Trace:
I20251026 14:59:17.241974 22731 rpcz_store.cc:276] 1026 14:59:15.824395 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:16.475442 (+651047us) service_pool.cc:225] Handling call
1026 14:59:16.504457 (+ 29015us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:16.777260 (+272803us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:17.234847 (+457587us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b5400076a54. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:17.235077 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":617447296}
I20251026 14:59:17.230356 22728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 780) took 1657 ms. Trace:
I20251026 14:59:17.243598 22728 rpcz_store.cc:276] 1026 14:59:15.573100 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:16.247026 (+673926us) service_pool.cc:225] Handling call
1026 14:59:16.247309 (+ 283us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:16.319814 (+ 72505us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:17.230095 (+910281us) spinlock_profiling.cc:243] Waited 66.3 ms on lock 0x7b540030b6d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:17.230295 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":152456448}
I20251026 14:59:17.228578 24481 consensus_queue.cc:237] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:17.262873 22624 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:17.265950 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:17.233246 24495 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:17.270538 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:17.242983 24479 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:17.254117 24519 raft_consensus.cc:697] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:17.194222 24345 consensus_queue.cc:260] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.286476 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:17.289602 22733 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:17.291237 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:17.248354 24377 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.299104 24455 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:17.302839 24516 log.cc:927] Time spent T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
I20251026 14:59:17.304177 22799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 907) took 1669 ms. Trace:
I20251026 14:59:17.304590 22799 rpcz_store.cc:276] 1026 14:59:15.634745 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:15.821914 (+187169us) service_pool.cc:225] Handling call
1026 14:59:15.822214 (+ 300us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:16.018480 (+196266us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:16.018514 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:16.950651 (+932137us) spinlock_profiling.cc:243] Waited 4.21 ms on lock 0x7b2c000e0388. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cfa5d9b1 00007fd8cfa6cfe2 00007fd8d1199701 00007fd8d115dfda 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22
1026 14:59:16.951009 (+ 358us) log.cc:844] Serialized 26 byte log entry
1026 14:59:17.064396 (+113387us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:17.064434 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:17.064503 (+ 69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:17.303842 (+239339us) raft_consensus.cc:1685] finished
1026 14:59:17.303869 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:17.304108 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":16127,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":5832,"spinlock_wait_cycles":9681664,"thread_start_us":278757,"threads_started":2,"wal-append.queue_time_us":186526}
I20251026 14:59:17.307372 24455 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:17.309046 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1185) took 1556 ms. Trace:
I20251026 14:59:17.309492 22726 rpcz_store.cc:276] 1026 14:59:15.752424 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:16.352652 (+600228us) service_pool.cc:225] Handling call
1026 14:59:16.365251 (+ 12599us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:16.493755 (+128504us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:17.308786 (+815031us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b54002dcb54. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:17.308978 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":293530496}
I20251026 14:59:17.312553 24341 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:17.315228 22700 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:17.302667 22729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 746) took 1843 ms. Trace:
I20251026 14:59:17.315871 22729 rpcz_store.cc:276] 1026 14:59:15.459775 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:15.461170 (+ 1395us) service_pool.cc:225] Handling call
1026 14:59:15.483495 (+ 22325us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:15.484009 (+ 514us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:15.484037 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:15.489817 (+ 5780us) log.cc:844] Serialized 52 byte log entry
1026 14:59:15.758987 (+269170us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:15.776477 (+ 17490us) log.cc:844] Serialized 14 byte log entry
1026 14:59:15.779938 (+ 3461us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:15.780017 (+ 79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:17.109663 (+1329646us) spinlock_profiling.cc:243] Waited 39.1 ms on lock 0x7b440002304c. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8cdfc0905 00007fd8cdfc1322 00007fd8d0d0a231 00007fd8d0d05b5a 00007fd8d0cf991e 00007fd8d0cf8c66 00007fd8d0cf8888 00007fd8d0d022c1 00007fd8d0d02279 00007fd8d0d02209
1026 14:59:17.302331 (+192668us) raft_consensus.cc:1685] finished
1026 14:59:17.302448 (+ 117us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:17.302610 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1843,"prepare.queue_time_us":16998,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":2046,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":3921,"spinlock_wait_cycles":90033920,"thread_start_us":214275,"threads_started":1,"wal-append.queue_time_us":321693}
I20251026 14:59:17.293730 24495 raft_consensus.cc:2804] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:17.313596 24519 consensus_queue.cc:237] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.276662 24479 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:17.297859 24352 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:17.312948 24332 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.328260 24031 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.323529 24495 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:17.347730 22700 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:17.349618 24115 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:17.345556 24495 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:17.364079 22654 raft_consensus.cc:1240] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251026 14:59:17.337656 24294 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:17.394249 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 131469us
I20251026 14:59:17.402565 22730 raft_consensus.cc:2393] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 4: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:17.402506 22624 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:17.384716 24353 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.397032 24325 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:17.415721 24519 raft_consensus.cc:2804] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:17.416325 24519 raft_consensus.cc:697] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:17.436503 22656 raft_consensus.cc:1240] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:17.412555 24490 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:17.442978 24479 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
W20251026 14:59:17.444573 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:17.414680 24377 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:17.447631 22808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49652 (request call id 149) took 1140 ms. Trace:
W20251026 14:59:17.450901 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:17.459160 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 629957us
I20251026 14:59:17.505055 22773 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:17.461824 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:17.517360 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 67594.7us
W20251026 14:59:17.519290 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 52741.6us
I20251026 14:59:17.465852 22808 rpcz_store.cc:276] 1026 14:59:16.307437 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:16.819669 (+512232us) service_pool.cc:225] Handling call
1026 14:59:16.822241 (+ 2572us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:16.827751 (+ 5510us) raft_consensus.cc:1534] Early marking committed up to index 6
1026 14:59:16.840189 (+ 12438us) log.cc:844] Serialized 14 byte log entry
1026 14:59:17.447135 (+606946us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b18002cd280. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce026caa 00007fd8d7698a29 00007fd8cdee6c30 00007fd8ce0aea94 00007fd8ce0aea05 00007fd8d1155c00 00007fd8d11682a6 00007fd8d1178e17 00007fd8d1178d98
1026 14:59:17.447293 (+ 158us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:17.447346 (+ 53us) raft_consensus.cc:1645] Marking committed up to 6
1026 14:59:17.447388 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:17.447454 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:17.447577 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2220,"prepare.queue_time_us":14389,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"spinlock_wait_cycles":43147648}
I20251026 14:59:17.524331 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:17.525718 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:17.526355 22803 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:17.526736 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:17.527302 22808 raft_consensus.cc:2468] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:59:17.528784 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:17.533530 22799 raft_consensus.cc:1240] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
W20251026 14:59:17.534420 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:17.495820 24352 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:17.415077 24325 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.529610 22658 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:17.607520 22700 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:17.608311 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:17.608901 22808 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:17.612346 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:17.612998 22803 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:17.524279 24341 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:17.531018 24332 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:17.528198 24479 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:17.418116 24519 consensus_queue.cc:237] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.650550 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:17.613659 24476 raft_consensus.cc:2804] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:17.610247 24031 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:17.614601 24322 raft_consensus.cc:2804] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election won for term 4
W20251026 14:59:17.654029 24530 log.cc:927] Time spent T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.226s user 0.001s sys 0.000s
W20251026 14:59:17.657402 24538 log.cc:927] Time spent T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1: Append to log took a long time: real 0.222s user 0.001s sys 0.000s
I20251026 14:59:17.654762 24502 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:17.417968 24536 consensus_queue.cc:1059] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.202s
I20251026 14:59:17.535419 24506 consensus_queue.cc:1048] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.087s
W20251026 14:59:17.653568 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 148597us
I20251026 14:59:17.662288 22801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 979) took 2082 ms. Trace:
I20251026 14:59:17.663983 22773 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:17.663746 22801 rpcz_store.cc:276] 1026 14:59:15.580158 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:15.776703 (+196545us) service_pool.cc:225] Handling call
1026 14:59:16.002268 (+225565us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:16.056831 (+ 54563us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:17.662073 (+1605242us) spinlock_profiling.cc:243] Waited 225 ms on lock 0x7b540003a7d0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:17.662227 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":517892352}
W20251026 14:59:17.666910 24535 log.cc:927] Time spent T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.127s user 0.000s sys 0.001s
I20251026 14:59:17.669451 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:17.670047 22731 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:17.671828 24495 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:17.675750 24524 log.cc:927] Time spent T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
W20251026 14:59:17.675942 24533 log.cc:927] Time spent T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.235s user 0.001s sys 0.000s
I20251026 14:59:17.677942 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:17.678558 22732 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:17.527307 24515 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:59:17.611759 24352 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.680411 22773 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:17.618114 24341 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:17.688480 24531 log.cc:927] Time spent T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.282s user 0.001s sys 0.000s
I20251026 14:59:17.656229 24476 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:17.658416 24502 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.665210 24294 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:17.629473 24479 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.694306 22808 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:17.694864 22773 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:17.674923 24322 raft_consensus.cc:697] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:17.698113 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:17.699719 22731 raft_consensus.cc:2468] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:17.701050 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:17.689333 24405 raft_consensus.cc:2749] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:17.701884 22731 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:17.702303 24405 raft_consensus.cc:2804] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:17.702908 24405 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:17.703403 24405 raft_consensus.cc:3060] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:17.702265 22732 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:17.711433 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:17.711846 24224 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:17.712323 22732 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:17.712682 24224 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:17.684198 24079 raft_consensus.cc:2804] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:17.721897 22803 pending_rounds.cc:85] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821: Aborting all ops after (but not including) 1
I20251026 14:59:17.722265 22803 pending_rounds.cc:107] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20251026 14:59:17.722601 22803 raft_consensus.cc:2889] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:17.723822 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:17.722059 24224 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:17.693617 24476 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:17.718575 24422 consensus_queue.cc:1048] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20251026 14:59:17.760146 22731 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
W20251026 14:59:17.788060 22700 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:17.726253 24079 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:17.732342 24353 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:17.716706 24455 consensus_queue.cc:1048] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.032s
W20251026 14:59:17.824564 24539 log.cc:927] Time spent T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.142s user 0.000s sys 0.001s
W20251026 14:59:17.821671 24531 log.cc:927] Time spent T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20251026 14:59:17.825596 24499 raft_consensus.cc:2804] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:17.826071 24499 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:17.826443 24499 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:17.748006 24325 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:17.821734 24079 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:17.831630 22804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 911) took 2123 ms. Trace:
I20251026 14:59:17.832445 24341 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:17.817507 24237 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:17.818432 24515 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:17.818943 24479 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:17.819623 24352 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:17.820147 24384 raft_consensus.cc:3055] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:17.819584 24322 consensus_queue.cc:237] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.720285 24115 consensus_queue.cc:1048] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20251026 14:59:17.693722 24377 consensus_queue.cc:1048] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.052s
I20251026 14:59:17.832062 22804 rpcz_store.cc:276] 1026 14:59:15.708353 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:16.060954 (+352601us) service_pool.cc:225] Handling call
1026 14:59:16.061235 (+ 281us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:16.263315 (+202080us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:17.831360 (+1568045us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b54001b8fd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:17.831564 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":439878016}
I20251026 14:59:17.818859 24426 raft_consensus.cc:3055] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:17.815126 24495 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:17.861392 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:17.863097 24479 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:17.863716 24479 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.866246 24479 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:17.854456 24384 raft_consensus.cc:740] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:17.880720 22774 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:17.882164 22726 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.004s sys 0.009s
W20251026 14:59:17.882484 22726 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.567s user 0.004s sys 0.009s
I20251026 14:59:17.882361 22774 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:17.858486 24426 raft_consensus.cc:740] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:17.888099 22807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 918) took 1928 ms. Trace:
I20251026 14:59:17.890638 22774 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:17.891786 22804 raft_consensus.cc:1240] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:17.889627 22807 rpcz_store.cc:276] 1026 14:59:15.961850 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:16.152652 (+190802us) service_pool.cc:225] Handling call
1026 14:59:16.153047 (+ 395us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:16.288358 (+135311us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:16.288406 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:16.300947 (+ 12541us) log.cc:844] Serialized 26 byte log entry
1026 14:59:16.456259 (+155312us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:16.456333 (+ 74us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:17.880814 (+1424481us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b54001bdd54. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:17.880865 (+ 51us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:17.880913 (+ 48us) raft_consensus.cc:1685] finished
1026 14:59:17.880925 (+ 12us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:17.888012 (+ 7087us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10785,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":3248,"spinlock_wait_cycles":310147968,"thread_start_us":141040,"threads_started":1,"wal-append.queue_time_us":187194,"wal-append.run_cpu_time_us":1971,"wal-append.run_wall_time_us":1114925}
I20251026 14:59:17.885951 24455 consensus_queue.cc:1048] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20251026 14:59:17.904129 24228 raft_consensus.cc:2804] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:17.886941 24377 raft_consensus.cc:2749] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251026 14:59:17.911401 22700 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:17.934832 24543 log.cc:927] Time spent T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20251026 14:59:17.921416 24426 consensus_queue.cc:260] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.944746 24426 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:17.925848 24541 log.cc:927] Time spent T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
I20251026 14:59:17.915685 24228 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:17.948297 24228 raft_consensus.cc:3060] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:17.906785 24115 raft_consensus.cc:2804] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:17.949772 24115 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:17.950290 24115 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:17.941702 24502 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:17.954968 24494 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:17.965996 24079 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:17.967379 22604 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 37e63cca2eb94ba8ac9b6c8cdf99c4a5, TS 0d684445287b450a8f738b6ad4b59821 -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:17.967832 22604 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 37e63cca2eb94ba8ac9b6c8cdf99c4a5, TS 0d684445287b450a8f738b6ad4b59821 -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.970297 24079 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:17.973578 24405 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.974918 22808 raft_consensus.cc:1240] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:17.975814 24405 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:17.943753 24476 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:18.000538 22654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1181) took 1115 ms. Trace:
I20251026 14:59:18.001178 22654 rpcz_store.cc:276] 1026 14:59:16.863790 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:16.864154 (+ 364us) service_pool.cc:225] Handling call
1026 14:59:16.864520 (+ 366us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:17.364702 (+500182us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:17.979242 (+614540us) spinlock_profiling.cc:243] Waited 499 ms on lock 0x7b54002a85d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:17.979458 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1148071680}
I20251026 14:59:17.915213 24384 consensus_queue.cc:260] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:17.955822 24494 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:18.015383 24479 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:18.047096 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 445246us
I20251026 14:59:18.005107 24499 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:18.058357 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 361612us
I20251026 14:59:18.060412 22773 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:18.060626 24327 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:18.062929 24327 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:18.065093 24544 raft_consensus.cc:2749] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:18.065379 24554 consensus_queue.cc:1048] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:18.065899 24327 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:18.061772 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:18.066908 24544 consensus_queue.cc:1048] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:59:18.065176 24557 consensus_queue.cc:1048] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20251026 14:59:18.069151 22773 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:18.051630 24479 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:18.086932 24479 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:18.088743 24499 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:18.090426 24406 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:18.090937 24406 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:18.062850 24115 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:18.095356 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 802) took 1143 ms. Trace:
I20251026 14:59:18.095786 22726 rpcz_store.cc:276] 1026 14:59:16.951998 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:17.310415 (+358417us) service_pool.cc:225] Handling call
1026 14:59:17.310779 (+ 364us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:17.311180 (+ 401us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:17.311209 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:17.311664 (+ 455us) log.cc:844] Serialized 26 byte log entry
1026 14:59:17.882949 (+571285us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:17.900301 (+ 17352us) log.cc:844] Serialized 14 byte log entry
1026 14:59:17.900627 (+ 326us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:17.900736 (+ 109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:18.095113 (+194377us) raft_consensus.cc:1685] finished
1026 14:59:18.095131 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:18.095283 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":17086,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":718,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":5347,"thread_start_us":567711,"threads_started":1,"wal-append.queue_time_us":778012}
W20251026 14:59:18.106088 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 213976us
I20251026 14:59:18.044441 24384 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:18.107997 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:18.113602 22726 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:18.114559 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:18.115306 22726 raft_consensus.cc:2393] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 5: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:18.115182 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:17.941725 24403 consensus_queue.cc:260] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:18.109110 22805 raft_consensus.cc:1240] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:18.110293 24499 consensus_queue.cc:1048] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:59:18.112380 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:18.112319 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:18.122818 22807 raft_consensus.cc:1240] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:18.116757 22773 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:17.973043 24228 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:18.118592 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:59:18.120774 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:18.126020 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:18.127012 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:18.106222 24479 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:18.107997 24544 raft_consensus.cc:2804] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:18.127099 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:18.124490 22773 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:18.128273 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:18.129958 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:18.157575 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 787) took 1681 ms. Trace:
I20251026 14:59:18.131563 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:18.157950 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:18.131834 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:18.158025 22734 rpcz_store.cc:276] 1026 14:59:16.476415 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:16.789246 (+312831us) service_pool.cc:225] Handling call
1026 14:59:16.789659 (+ 413us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:16.833576 (+ 43917us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:16.833625 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:16.833690 (+ 65us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:16.833741 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:18.157297 (+1323556us) spinlock_profiling.cc:243] Waited 43.6 ms on lock 0x7b54003b38d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:18.157355 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:18.157503 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":100241280}
I20251026 14:59:18.159770 22801 raft_consensus.cc:2393] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:18.159863 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:18.160646 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:18.161931 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:18.108271 24557 consensus_queue.cc:1048] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
W20251026 14:59:18.185026 22700 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:18.202764 22807 raft_consensus.cc:3060] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
W20251026 14:59:18.131420 24553 log.cc:927] Time spent T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.067s user 0.000s sys 0.001s
W20251026 14:59:18.203866 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 79457.8us
I20251026 14:59:18.213207 22773 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:18.214928 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:18.215050 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:18.215916 22799 raft_consensus.cc:2468] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:18.133033 24403 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:18.225548 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:18.130633 24554 raft_consensus.cc:2804] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251026 14:59:18.227195 24561 log.cc:927] Time spent T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.159s user 0.000s sys 0.001s
I20251026 14:59:18.202358 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:18.234018 22801 raft_consensus.cc:1240] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20251026 14:59:18.232683 22727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 812) took 1049 ms. Trace:
I20251026 14:59:18.234623 22807 raft_consensus.cc:2468] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:18.212139 24405 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:18.236140 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:18.242514 22727 rpcz_store.cc:276] 1026 14:59:17.183296 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:17.941591 (+758295us) service_pool.cc:225] Handling call
1026 14:59:17.941971 (+ 380us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:17.942345 (+ 374us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:17.942376 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:17.942987 (+ 611us) log.cc:844] Serialized 52 byte log entry
1026 14:59:18.096776 (+153789us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:18.126210 (+ 29434us) log.cc:844] Serialized 14 byte log entry
1026 14:59:18.126488 (+ 278us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:18.126600 (+ 112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:18.232435 (+105835us) raft_consensus.cc:1685] finished
1026 14:59:18.232454 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:18.232614 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":29894,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":4520,"raft.queue_time_us":4581,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":27724,"thread_start_us":153376,"threads_started":1,"wal-append.queue_time_us":214435}
I20251026 14:59:18.231771 24314 raft_consensus.cc:2749] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:18.159078 24479 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:18.224941 24115 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:18.204327 24544 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:18.235008 24463 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:18.240835 24554 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:18.252580 22731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 809) took 1071 ms. Trace:
I20251026 14:59:18.250088 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:18.240613 24476 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Requested vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:59:18.261204 22656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1187) took 1282 ms. Trace:
I20251026 14:59:18.261442 22727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 818) took 1059 ms. Trace:
I20251026 14:59:18.262007 22731 rpcz_store.cc:276] 1026 14:59:17.181211 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:17.770099 (+588888us) service_pool.cc:225] Handling call
1026 14:59:17.770582 (+ 483us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:17.773148 (+ 2566us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:17.773192 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:17.773764 (+ 572us) log.cc:844] Serialized 26 byte log entry
1026 14:59:18.095081 (+321317us) spinlock_profiling.cc:243] Waited 36.9 ms on lock 0x7b1800362860. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce0942f7 00007fd8d6e6d219 00007fd8ce0b0c4b 00007fd8ce0aeeef 00007fd8ce0b11b2 00007fd8d0cf82bb 00007fd8d0cfddc3 00007fd8d0cfe171 00007fd8d111965f
1026 14:59:18.246466 (+151385us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:18.246512 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:18.246591 (+ 79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:18.247891 (+ 1300us) raft_consensus.cc:1685] finished
1026 14:59:18.247916 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:18.252491 (+ 4575us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2070,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":11555,"spinlock_wait_cycles":84848384,"thread_start_us":151192,"threads_started":1,"wal-append.queue_time_us":472678}
I20251026 14:59:18.262938 24494 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:18.263371 22725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 1189) took 1784 ms (client timeout 2135 ms). Trace:
I20251026 14:59:18.265317 22656 rpcz_store.cc:276] 1026 14:59:16.978886 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:16.979290 (+ 404us) service_pool.cc:225] Handling call
1026 14:59:16.979696 (+ 406us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:17.436998 (+457302us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:18.260961 (+823963us) spinlock_profiling.cc:243] Waited 457 ms on lock 0x7b54001d97d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:18.261141 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1050077056}
I20251026 14:59:18.265379 22727 rpcz_store.cc:276] 1026 14:59:17.202448 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:18.248715 (+1046267us) service_pool.cc:225] Handling call
1026 14:59:18.260907 (+ 12192us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:18.261099 (+ 192us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:18.261124 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:18.261166 (+ 42us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:18.261195 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:18.261251 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:18.261392 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:18.662365 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:18.494125 22655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 981) took 1153 ms. Trace:
I20251026 14:59:18.343657 22653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 973) took 1401 ms. Trace:
I20251026 14:59:18.258246 24544 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
W20251026 14:59:18.431648 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 131823us
I20251026 14:59:18.706823 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:18.718125 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:18.345001 22657 raft_consensus.cc:1240] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20251026 14:59:18.784447 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:18.269661 22700 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:18.822914 22654 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
W20251026 14:59:18.551026 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 871762us
I20251026 14:59:18.491106 22650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1191) took 1462 ms. Trace:
I20251026 14:59:18.719755 22729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 803) took 1767 ms. Trace:
I20251026 14:59:18.490437 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:18.289667 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 807) took 1109 ms. Trace:
I20251026 14:59:18.824818 22650 rpcz_store.cc:276] 1026 14:59:17.029150 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:17.059287 (+ 30137us) service_pool.cc:225] Handling call
1026 14:59:17.069461 (+ 10174us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:17.134025 (+ 64564us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:18.490787 (+1356762us) spinlock_profiling.cc:243] Waited 63.8 ms on lock 0x7b540010d6d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:18.491032 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":146728704}
W20251026 14:59:18.521425 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 1.00289e+06us
W20251026 14:59:18.267669 22725 rpcz_store.cc:269] 1026 14:59:16.479239 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:16.658985 (+179746us) service_pool.cc:225] Handling call
1026 14:59:18.263168 (+1604183us) spinlock_profiling.cc:243] Waited 296 ms on lock 0x7b54003b33d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:18.263319 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":681605376}
I20251026 14:59:18.794452 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:18.818244 22805 raft_consensus.cc:2410] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:18.774446 22697 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:18.802227 22726 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.044s sys 0.093s
W20251026 14:59:18.831295 22726 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.572s user 0.045s sys 0.094s
I20251026 14:59:18.703629 22655 rpcz_store.cc:276] 1026 14:59:17.340945 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:17.341121 (+ 176us) service_pool.cc:225] Handling call
1026 14:59:17.341556 (+ 435us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:17.365411 (+ 23855us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:17.365462 (+ 51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:17.392325 (+ 26863us) log.cc:844] Serialized 52 byte log entry
1026 14:59:17.533226 (+140901us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:17.621526 (+ 88300us) log.cc:844] Serialized 14 byte log entry
1026 14:59:18.493331 (+871805us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b18000b6980. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce026caa 00007fd8d7698a29 00007fd8cdee6c30 00007fd8ce0aea94 00007fd8ce0aea05 00007fd8d1155c00 00007fd8d11682a6 00007fd8d1178e17 00007fd8d1178d98
1026 14:59:18.493581 (+ 250us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:18.493695 (+ 114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:18.493759 (+ 64us) raft_consensus.cc:1685] finished
1026 14:59:18.493871 (+ 112us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:18.494047 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":155634,"prepare.queue_time_us":82139,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":2319,"raft.queue_time_us":893033,"raft.run_cpu_time_us":337,"raft.run_wall_time_us":16087,"spinlock_wait_cycles":30053248,"thread_start_us":129157,"threads_started":1,"wal-append.queue_time_us":416070}
W20251026 14:59:18.451520 24553 log.cc:927] Time spent T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
I20251026 14:59:18.819650 24377 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:18.703809 22653 rpcz_store.cc:276] 1026 14:59:16.942750 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:16.942876 (+ 126us) service_pool.cc:225] Handling call
1026 14:59:18.343300 (+1400424us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b54000dc354. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:18.343565 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":311532416}
I20251026 14:59:18.525342 24536 raft_consensus.cc:3055] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 14:59:18.825359 22729 rpcz_store.cc:276] 1026 14:59:16.952304 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:17.317252 (+364948us) service_pool.cc:225] Handling call
1026 14:59:17.441520 (+124268us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:17.441890 (+ 370us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:17.441923 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:17.442371 (+ 448us) log.cc:844] Serialized 26 byte log entry
1026 14:59:17.608404 (+166033us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:17.608451 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:17.608513 (+ 62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:17.935894 (+327381us) raft_consensus.cc:1685] finished
1026 14:59:17.935918 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:18.719328 (+783410us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b54003b2ed0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:18.719671 (+ 343us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":270,"raft.run_cpu_time_us":71,"raft.run_wall_time_us":70,"spinlock_wait_cycles":259599744,"thread_start_us":165741,"threads_started":1,"wal-append.queue_time_us":400251}
I20251026 14:59:18.705089 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:18.755338 24224 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:18.261488 24463 raft_consensus.cc:3060] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:18.701684 24569 log.cc:927] Time spent T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.179s user 0.001s sys 0.000s
I20251026 14:59:18.849175 24536 raft_consensus.cc:740] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:18.267130 22802 raft_consensus.cc:2393] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:18.831861 22657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1188) took 1820 ms. Trace:
I20251026 14:59:18.495537 24406 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:18.464627 24322 raft_consensus.cc:1064] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821: attempting to promote NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 to VOTER
I20251026 14:59:18.839705 24476 raft_consensus.cc:2749] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251026 14:59:18.496835 24567 log.cc:927] Time spent T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.119s user 0.001s sys 0.000s
W20251026 14:59:18.296387 24563 log.cc:927] Time spent T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
W20251026 14:59:18.295269 24552 log.cc:927] Time spent T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20251026 14:59:18.835044 22624 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:18.704952 22728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 821) took 1245 ms. Trace:
I20251026 14:59:18.875612 22728 rpcz_store.cc:276] 1026 14:59:17.459861 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:18.319982 (+860121us) service_pool.cc:225] Handling call
1026 14:59:18.393527 (+ 73545us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:18.393789 (+ 262us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:18.393828 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:18.393887 (+ 59us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:18.393931 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:18.394002 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:18.704879 (+310877us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:18.819882 24566 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.021s sys 0.019s
W20251026 14:59:18.879307 24566 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.584s user 0.022s sys 0.019s
I20251026 14:59:18.879508 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:18.743784 24345 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:18.267046 24554 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:18.871592 24322 raft_consensus.cc:1068] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821: Unable to promote non-voter 248493ec312e4398b828c6cb00e5c4b1: Illegal state: Leader has not yet committed an operation in its own term
I20251026 14:59:18.883985 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:18.386425 22731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47996 (request call id 103) took 1149 ms. Trace:
I20251026 14:59:18.884608 22655 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:18.721387 24418 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:18.656451 24518 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:18.267498 24308 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:18.300513 24310 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:18.847851 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:18.839208 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 65031.9us
I20251026 14:59:18.839752 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:18.706269 24479 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:18.849753 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:18.850384 22656 raft_consensus.cc:3060] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:18.895433 22697 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:18.896461 22805 raft_consensus.cc:2393] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:18.896814 22697 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:18.834467 24381 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:18.772130 24228 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:18.897693 22622 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:18.825973 22732 rpcz_store.cc:276] 1026 14:59:17.180134 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:17.727060 (+546926us) service_pool.cc:225] Handling call
1026 14:59:17.731390 (+ 4330us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:17.734991 (+ 3601us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:17.735054 (+ 63us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:17.739192 (+ 4138us) log.cc:844] Serialized 52 byte log entry
1026 14:59:18.064278 (+325086us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:18.064902 (+ 624us) log.cc:844] Serialized 14 byte log entry
1026 14:59:18.065796 (+ 894us) log.cc:844] Serialized 14 byte log entry
1026 14:59:18.066163 (+ 367us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:18.066233 (+ 70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:18.289333 (+223100us) raft_consensus.cc:1685] finished
1026 14:59:18.289368 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:18.289579 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2967,"prepare.queue_time_us":219,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":65,"raft.queue_time_us":3260,"raft.run_cpu_time_us":334,"raft.run_wall_time_us":11489,"thread_start_us":204222,"threads_started":1,"wal-append.queue_time_us":249192}
I20251026 14:59:18.858848 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:18.898947 22729 raft_consensus.cc:2368] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4.
I20251026 14:59:18.829331 24309 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:18.857538 22657 rpcz_store.cc:276] 1026 14:59:17.011378 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:17.011551 (+ 173us) service_pool.cc:225] Handling call
1026 14:59:18.344663 (+1333112us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:18.818696 (+474033us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:18.831679 (+ 12983us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54000dc350. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:18.831807 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3064932736}
I20251026 14:59:18.774056 24478 consensus_queue.cc:1059] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.078s
I20251026 14:59:18.837523 24333 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:18.835664 24401 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:18.838348 24466 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:18.866541 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:18.912926 22653 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
W20251026 14:59:18.868330 24405 consensus_meta.cc:287] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.619s user 0.007s sys 0.004s
I20251026 14:59:18.855532 24536 consensus_queue.cc:260] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:18.849800 24377 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:18.884934 22731 rpcz_store.cc:276] 1026 14:59:17.237390 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:18.266536 (+1029146us) service_pool.cc:225] Handling call
1026 14:59:18.385750 (+119214us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:18.385980 (+ 230us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:18.386017 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:18.386076 (+ 59us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:18.386121 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:18.386193 (+ 72us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:18.386346 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:18.869046 24336 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:18.891140 22728 raft_consensus.cc:1240] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:18.922848 22728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 826) took 1461 ms. Trace:
I20251026 14:59:18.923192 22728 rpcz_store.cc:276] 1026 14:59:17.461645 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:18.876707 (+1415062us) service_pool.cc:225] Handling call
1026 14:59:18.890967 (+ 14260us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:18.922629 (+ 31662us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:18.922804 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:18.913800 24401 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:18.895475 22650 raft_consensus.cc:2468] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:18.924876 24336 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:18.925817 24401 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:18.913829 24411 raft_consensus.cc:2804] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:18.926599 24411 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:18.926975 24411 raft_consensus.cc:3060] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:18.826195 22808 raft_consensus.cc:2393] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:18.707742 24142 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:18.927898 24142 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:18.921432 24466 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:18.930102 24142 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:18.930681 24142 raft_consensus.cc:493] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:18.931231 24142 raft_consensus.cc:515] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:18.931845 24115 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:18.858055 24224 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:18.934785 24224 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:18.935070 24377 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:18.937075 22799 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 1.
I20251026 14:59:18.903918 24476 raft_consensus.cc:2749] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:18.900593 22773 leader_election.cc:400] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Vote denied by peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507) with higher term. Message: Invalid argument: T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4.
I20251026 14:59:18.944170 22773 leader_election.cc:403] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
W20251026 14:59:18.944717 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 44268.7us
I20251026 14:59:18.909310 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:18.910097 22729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 827) took 1448 ms. Trace:
I20251026 14:59:18.909308 22622 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:18.906028 24354 raft_consensus.cc:493] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:18.295423 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:18.949482 22801 raft_consensus.cc:2468] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:18.950897 24142 leader_election.cc:290] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:18.914196 24527 raft_consensus.cc:2804] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:18.897485 24308 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:18.894747 24518 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:18.932482 24115 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:18.877251 24397 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:18.900305 24479 raft_consensus.cc:2749] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:18.899627 22697 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:18.942045 22733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 801) took 1990 ms. Trace:
W20251026 14:59:18.961468 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 61901.8us
W20251026 14:59:18.908493 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 73534.2us
I20251026 14:59:18.961382 22733 rpcz_store.cc:276] 1026 14:59:16.951685 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:17.292475 (+340790us) service_pool.cc:225] Handling call
1026 14:59:17.293024 (+ 549us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:17.293561 (+ 537us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:17.293605 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:17.301327 (+ 7722us) log.cc:844] Serialized 52 byte log entry
1026 14:59:18.094571 (+793244us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b1800413860. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce0942f7 00007fd8d6e6d219 00007fd8ce0b0c4b 00007fd8ce0aeeef 00007fd8ce0b11b2 00007fd8d0cf82bb 00007fd8d0cfddc3 00007fd8d0cfe171 00007fd8d111965f
1026 14:59:18.239897 (+145326us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:18.239949 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:18.240042 (+ 93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:18.922735 (+682693us) raft_consensus.cc:1685] finished
1026 14:59:18.922756 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:18.941950 (+ 19194us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":124,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":5466,"spinlock_wait_cycles":24086272,"thread_start_us":145142,"threads_started":1,"wal-append.queue_time_us":1041536}
I20251026 14:59:18.903437 24381 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:18.952104 22622 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:18.953790 22650 raft_consensus.cc:1240] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:18.898171 24310 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:18.921693 24405 raft_consensus.cc:2749] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:18.958571 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 817) took 1772 ms. Trace:
I20251026 14:59:18.959656 22803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 955) took 1272 ms. Trace:
I20251026 14:59:18.959010 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:18.962519 22624 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:18.946825 22729 rpcz_store.cc:276] 1026 14:59:17.461959 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:18.899892 (+1437933us) service_pool.cc:225] Handling call
1026 14:59:18.909614 (+ 9722us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:18.909786 (+ 172us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:18.909810 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:18.909851 (+ 41us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:18.909880 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:18.909930 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:18.910041 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:18.963409 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:18.913951 24333 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:18.955440 24466 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:18.905990 24309 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:18.962096 22731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47996 (request call id 104) took 1230 ms. Trace:
I20251026 14:59:19.001714 22731 rpcz_store.cc:276] 1026 14:59:17.731337 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:18.922570 (+1191233us) service_pool.cc:225] Handling call
1026 14:59:18.961633 (+ 39063us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:18.961811 (+ 178us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:18.961835 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:18.961875 (+ 40us) raft_consensus.cc:1645] Marking committed up to 6
1026 14:59:18.961900 (+ 25us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:18.961950 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:18.962046 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:18.920383 22656 raft_consensus.cc:2468] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:18.959607 24527 raft_consensus.cc:697] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:18.958068 24322 raft_consensus.cc:1064] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821: attempting to promote NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 to VOTER
I20251026 14:59:18.979506 22653 raft_consensus.cc:2468] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:18.979221 22803 rpcz_store.cc:276] 1026 14:59:17.687805 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:17.688040 (+ 235us) service_pool.cc:225] Handling call
1026 14:59:17.721678 (+ 33638us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:17.736495 (+ 14817us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:17.736533 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:17.739841 (+ 3308us) log.cc:844] Serialized 52 byte log entry
1026 14:59:17.979345 (+239504us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:17.979383 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:17.979448 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:18.959441 (+979993us) raft_consensus.cc:1685] finished
1026 14:59:18.959465 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:18.959599 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":129701,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":110,"thread_start_us":112256,"threads_started":1,"wal-append.queue_time_us":312154}
I20251026 14:59:18.963903 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:18.980675 22622 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:19.009603 22732 raft_consensus.cc:2468] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:18.978179 22726 rpcz_store.cc:276] 1026 14:59:17.186116 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:18.211974 (+1025858us) service_pool.cc:225] Handling call
1026 14:59:18.212355 (+ 381us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:18.223413 (+ 11058us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:18.223523 (+ 110us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:18.234688 (+ 11165us) log.cc:844] Serialized 52 byte log entry
1026 14:59:18.831760 (+597072us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:18.831799 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:18.831867 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:18.958358 (+126491us) raft_consensus.cc:1685] finished
1026 14:59:18.958380 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:18.958514 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":495,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":9638,"thread_start_us":572365,"threads_started":1,"wal-append.queue_time_us":635362}
I20251026 14:59:18.982081 24031 raft_consensus.cc:2804] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:18.967571 24479 raft_consensus.cc:2804] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:18.966749 24397 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:18.991283 24308 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:18.991824 24570 log.cc:927] Time spent T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.113s user 0.000s sys 0.001s
I20251026 14:59:18.994946 22799 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "11ae57d90c614ce2993629f95b777120"
dest_uuid: "0d684445287b450a8f738b6ad4b59821"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:44012
I20251026 14:59:19.020340 22799 raft_consensus.cc:606] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:19.020677 22799 raft_consensus.cc:614] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251026 14:59:18.950142 24377 raft_consensus.cc:2749] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507) with higher term. Message: Invalid argument: T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4.
I20251026 14:59:18.951452 24142 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:18.999133 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:18.951579 24336 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:19.000061 22624 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:18.951697 24572 log.cc:927] Time spent T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.078s user 0.001s sys 0.000s
I20251026 14:59:18.963438 24476 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:18.954646 24536 raft_consensus.cc:3060] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:59:19.001031 24492 raft_consensus.cc:2804] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:18.953536 24354 raft_consensus.cc:515] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:19.012123 22773 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:19.001086 24403 consensus_meta.cc:287] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.768s user 0.008s sys 0.004s
I20251026 14:59:19.014487 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:19.017567 24479 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:19.045746 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.016281 22726 raft_consensus.cc:1275] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20251026 14:59:19.046504 22656 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:19.046861 22726 pending_rounds.cc:85] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e: Aborting all ops after (but not including) 5
I20251026 14:59:19.047297 22726 pending_rounds.cc:107] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20251026 14:59:19.047622 22726 raft_consensus.cc:2889] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:18.991287 24518 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:19.021719 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:19.019362 24322 raft_consensus.cc:1068] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821: Unable to promote non-voter 248493ec312e4398b828c6cb00e5c4b1: Illegal state: Leader has not yet committed an operation in its own term
I20251026 14:59:19.026938 22732 raft_consensus.cc:1275] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20251026 14:59:19.053714 22732 pending_rounds.cc:85] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: Aborting all ops after (but not including) 5
I20251026 14:59:19.054078 22732 pending_rounds.cc:107] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20251026 14:59:19.054379 22732 raft_consensus.cc:2889] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:19.005560 24332 raft_consensus.cc:2764] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20251026 14:59:19.021021 24406 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:19.014247 24103 raft_consensus.cc:2749] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:19.007180 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.018083 24031 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:19.087062 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.013051 24405 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:18.953622 24337 consensus_queue.cc:1059] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.066s
I20251026 14:59:19.043064 22729 raft_consensus.cc:1275] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:19.100567 22729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1198) took 1013 ms. Trace:
I20251026 14:59:19.080456 22726 raft_consensus.cc:1240] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:19.101823 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1204) took 1013 ms. Trace:
I20251026 14:59:19.102151 22726 rpcz_store.cc:276] 1026 14:59:18.088414 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:19.056081 (+967667us) service_pool.cc:225] Handling call
1026 14:59:19.080267 (+ 24186us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:19.101542 (+ 21275us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:19.101768 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:19.102558 22727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 822) took 1642 ms. Trace:
I20251026 14:59:19.103478 22726 raft_consensus.cc:1240] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20251026 14:59:19.104111 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1205) took 1015 ms. Trace:
I20251026 14:59:19.104379 22726 rpcz_store.cc:276] 1026 14:59:18.088676 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:19.102933 (+1014257us) service_pool.cc:225] Handling call
1026 14:59:19.103254 (+ 321us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:19.103892 (+ 638us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:19.104066 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:19.049741 24466 raft_consensus.cc:2804] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:19.043398 24377 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:19.103073 22727 rpcz_store.cc:276] 1026 14:59:17.460250 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:18.817354 (+1357104us) service_pool.cc:225] Handling call
1026 14:59:18.838371 (+ 21017us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:18.968162 (+129791us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:18.968210 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:18.968269 (+ 59us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:18.968315 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:19.102254 (+133939us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b54000b95d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:19.102321 (+ 67us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:19.102491 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":297957248}
I20251026 14:59:19.022581 24308 consensus_queue.cc:1059] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20251026 14:59:19.051137 24479 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:19.019944 24397 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.107045 22726 raft_consensus.cc:1240] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251026 14:59:19.100937 22729 rpcz_store.cc:276] 1026 14:59:18.086784 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:19.035747 (+948963us) service_pool.cc:225] Handling call
1026 14:59:19.038859 (+ 3112us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:19.100305 (+ 61446us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:19.100513 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:19.131951 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1206) took 1043 ms. Trace:
I20251026 14:59:19.033291 24325 consensus_queue.cc:1059] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20251026 14:59:19.081106 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:19.057356 24492 raft_consensus.cc:697] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:19.034991 24142 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.099934 24031 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:19.133620 22728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47996 (request call id 105) took 1267 ms. Trace:
I20251026 14:59:19.140350 22799 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "3ea63385804c40689502aefb2b695e78"
dest_uuid: "0d684445287b450a8f738b6ad4b59821"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:44012
I20251026 14:59:19.134048 22653 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
W20251026 14:59:19.135264 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 135294us
I20251026 14:59:19.044087 24476 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.138444 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.132326 22726 rpcz_store.cc:276] 1026 14:59:18.088972 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:19.106483 (+1017511us) service_pool.cc:225] Handling call
1026 14:59:19.106825 (+ 342us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:19.131683 (+ 24858us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:19.131899 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:19.142885 22650 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:19.111258 24032 raft_consensus.cc:3055] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:19.142843 22624 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:19.140991 22799 raft_consensus.cc:606] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:19.099249 24410 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:19.146018 22727 raft_consensus.cc:1240] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: [2.2-5.3]
I20251026 14:59:19.147877 22727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1207) took 1058 ms. Trace:
I20251026 14:59:19.140472 22728 rpcz_store.cc:276] 1026 14:59:17.866075 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:18.923922 (+1057847us) service_pool.cc:225] Handling call
1026 14:59:18.924236 (+ 314us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:19.084545 (+160309us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:19.084578 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:19.084631 (+ 53us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:19.084677 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:19.133300 (+ 48623us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b54002dbc54. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:19.133385 (+ 85us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:19.133552 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":368224000}
I20251026 14:59:19.109213 24466 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:19.148491 22727 rpcz_store.cc:276] 1026 14:59:18.089245 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:19.129669 (+1040424us) service_pool.cc:225] Handling call
1026 14:59:19.145770 (+ 16101us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:19.147534 (+ 1764us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:19.147808 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:19.127750 24401 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:19.157476 24455 consensus_queue.cc:1059] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:59:19.157789 24333 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:19.157863 24401 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.158763 24333 raft_consensus.cc:3055] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:19.159214 24333 raft_consensus.cc:740] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:19.160184 24401 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:19.153028 24310 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:19.153096 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.160194 24333 consensus_queue.cc:260] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.139688 24377 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.167692 24336 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:19.171895 24309 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:19.127573 24322 consensus_queue.cc:1059] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20251026 14:59:19.140232 24353 raft_consensus.cc:2804] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251026 14:59:19.175520 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:19.106091 24405 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.127738 24406 consensus_queue.cc:1059] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20251026 14:59:19.147490 24032 raft_consensus.cc:740] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:19.151481 24308 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:19.099071 24527 consensus_queue.cc:237] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.138751 24422 consensus_queue.cc:1048] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.035s
I20251026 14:59:19.180370 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:19.182678 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.175117 24381 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:19.185132 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.171037 24466 raft_consensus.cc:3060] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:19.164997 24410 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:19.177599 24333 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:19.198547 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.200412 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.201010 22650 raft_consensus.cc:2468] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:19.201498 24479 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.202651 22731 raft_consensus.cc:2368] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6.
I20251026 14:59:19.204136 24479 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:19.204264 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.178306 24353 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:19.206061 22700 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:19.211299 22700 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:19.212738 22700 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:19.172857 24228 consensus_queue.cc:1059] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.019s
W20251026 14:59:19.207618 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:19.218287 22730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 813) took 2033 ms. Trace:
I20251026 14:59:19.219089 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:19.215595 22700 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:19.177109 24544 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.223951 24354 leader_election.cc:290] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:19.224609 24142 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:19.196522 24308 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.227394 24308 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:19.228739 24476 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:19.216619 24325 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:19.230167 24325 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.232071 24325 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:19.235831 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 211118us
I20251026 14:59:19.238097 24440 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:19.238638 24440 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.240659 24397 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:19.241185 24397 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:19.241662 24397 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.241852 24422 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:19.242496 24422 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.243501 24397 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:19.204898 24309 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:19.244755 24309 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.245632 24228 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:19.246250 24228 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.246802 24440 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:19.248687 24228 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:19.251914 24224 raft_consensus.cc:3055] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:19.251803 24554 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.252462 24224 raft_consensus.cc:740] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:19.256282 22658 raft_consensus.cc:2468] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:59:19.219584 22730 rpcz_store.cc:276] 1026 14:59:17.185032 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:18.092346 (+907314us) service_pool.cc:225] Handling call
1026 14:59:18.092729 (+ 383us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:18.098114 (+ 5385us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:18.098145 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:18.108867 (+ 10722us) log.cc:844] Serialized 52 byte log entry
1026 14:59:18.309189 (+200322us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:18.830144 (+520955us) log.cc:844] Serialized 14 byte log entry
1026 14:59:18.838110 (+ 7966us) log.cc:844] Serialized 14 byte log entry
1026 14:59:18.855263 (+ 17153us) spinlock_profiling.cc:243] Waited 954 us on lock 0x7b1800365c20. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce0b1abf 00007fd8ce0b3d61 00007fd8ce0b3d19 00007fd8ce0b3ca9 00007fd8ce0b3c71 00007fd8ce0b2f6d 0000000000379254 0000000000379099 00007fd8ce094d1a
1026 14:59:19.177403 (+322140us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:19.180093 (+ 2690us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:19.218008 (+ 37915us) raft_consensus.cc:1685] finished
1026 14:59:19.218039 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:19.218215 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22583,"prepare.queue_time_us":182637,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":1035,"raft.queue_time_us":26449,"raft.run_cpu_time_us":339,"raft.run_wall_time_us":11627,"spinlock_wait_cycles":4073600,"thread_start_us":925116,"threads_started":3,"wal-append.queue_time_us":726436}
I20251026 14:59:19.244863 24422 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:19.274745 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 311866us
W20251026 14:59:19.275017 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 59530.8us
I20251026 14:59:19.278488 22697 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:19.214485 24353 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:19.291834 22700 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:19.246802 24309 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:19.263765 24312 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:19.296835 22697 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:19.275658 24228 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:19.308383 22697 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:19.309692 22700 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:19.313331 24384 consensus_meta.cc:287] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 1.199s user 0.013s sys 0.000s
I20251026 14:59:19.314144 24411 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.316897 24411 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:19.196827 24492 consensus_queue.cc:237] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.330348 24405 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:19.318030 24377 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:19.293656 24403 consensus_queue.cc:1059] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.026s
I20251026 14:59:19.319094 24544 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:19.319806 24309 raft_consensus.cc:2804] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:19.321389 24312 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.306511 24372 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:19.311805 24354 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:19.325876 24554 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:19.307317 24422 consensus_queue.cc:1059] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20251026 14:59:19.307615 24410 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:19.357815 24377 consensus_queue.cc:1059] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20251026 14:59:19.372483 24554 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:19.372687 24309 raft_consensus.cc:697] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:19.373210 24554 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.362439 24354 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.373775 24309 consensus_queue.cc:237] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.375701 24354 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:19.293334 24479 raft_consensus.cc:2804] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:19.361166 24153 raft_consensus.cc:3055] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:19.376334 24479 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:19.362478 24372 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:19.376742 24153 raft_consensus.cc:740] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:19.376893 24479 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:19.377641 24153 consensus_queue.cc:260] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.378263 24372 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:19.378736 24153 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:19.380024 24312 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:19.380746 24544 consensus_queue.cc:1059] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:59:19.329058 24031 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.329453 22733 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:19.381698 24544 raft_consensus.cc:3055] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:19.382056 24507 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:19.382892 24031 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:19.382695 24507 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.332584 22697 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:19.383698 22650 raft_consensus.cc:2368] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6.
I20251026 14:59:19.335873 24310 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:19.336830 22700 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:19.339890 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:19.386359 22653 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:19.330696 24466 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.334398 24228 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.326910 24478 raft_consensus.cc:2749] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:19.324958 24333 raft_consensus.cc:2804] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:19.291810 24476 raft_consensus.cc:2804] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:19.280752 24308 raft_consensus.cc:2749] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:19.375767 24554 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:19.390656 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 53905.7us
I20251026 14:59:19.389734 24578 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:19.391067 24310 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.382088 24544 raft_consensus.cc:740] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:19.399137 22700 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:19.390604 24354 raft_consensus.cc:2804] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:19.392757 24345 raft_consensus.cc:2804] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 14:59:19.412750 22700 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:19.414237 22700 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:19.416589 22700 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:19.419420 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.420217 22657 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:19.422693 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.425046 22653 raft_consensus.cc:2468] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:19.425144 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.394567 24333 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:19.394470 24476 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:19.428103 24308 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:19.405771 24478 raft_consensus.cc:2804] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:19.456189 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.458281 22655 raft_consensus.cc:2410] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251026 14:59:19.412032 24354 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:19.461913 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 293942us
I20251026 14:59:19.435640 24333 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:19.437578 24476 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:19.427219 24345 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:19.457587 24478 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:19.478901 24354 raft_consensus.cc:3060] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:19.494308 24345 raft_consensus.cc:3060] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:59:19.491020 24336 raft_consensus.cc:697] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:19.539885 24518 raft_consensus.cc:2804] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:19.522266 24478 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:19.541369 24422 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:19.541697 22731 raft_consensus.cc:1275] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:19.585156 22805 raft_consensus.cc:1240] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:19.542876 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.543701 24401 raft_consensus.cc:3055] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:19.547459 24032 consensus_queue.cc:260] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.549849 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 824) took 2089 ms. Trace:
I20251026 14:59:19.552512 22554 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50170 (request call id 59) took 6644 ms. Trace:
I20251026 14:59:19.561115 24103 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:19.597560 24103 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:19.598989 22803 raft_consensus.cc:1275] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:19.599596 24103 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:19.600175 22803 raft_consensus.cc:1240] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:19.603222 24586 raft_consensus.cc:2804] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:19.603816 24586 raft_consensus.cc:697] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:19.605660 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:19.606905 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:19.607703 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:19.608664 24354 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.610723 24354 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:19.564471 24397 raft_consensus.cc:3055] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:19.565697 24224 consensus_queue.cc:260] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.581858 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:19.586566 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:19.587316 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 816) took 2401 ms. Trace:
I20251026 14:59:19.592864 22806 raft_consensus.cc:1240] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:19.593446 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:19.626360 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:19.627413 22805 raft_consensus.cc:2393] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 6: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:19.593173 22554 rpcz_store.cc:276] 1026 14:59:12.907813 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:12.908004 (+ 191us) service_pool.cc:225] Handling call
1026 14:59:19.552449 (+6644445us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 14:59:17.369956 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 14:59:17.445503 (+ 75547us) write_op.cc:432] Acquiring schema lock in shared mode
1026 14:59:17.445551 (+ 48us) write_op.cc:435] Acquired schema lock
1026 14:59:17.445570 (+ 19us) tablet.cc:660] Decoding operations
1026 14:59:17.625822 (+180252us) write_op.cc:620] Acquiring the partition lock for write op
1026 14:59:17.653473 (+ 27651us) write_op.cc:641] Partition lock acquired for write op
1026 14:59:17.656846 (+ 3373us) tablet.cc:683] Acquiring locks for 4 operations
1026 14:59:17.730477 (+ 73631us) tablet.cc:699] Row locks acquired
1026 14:59:17.733161 (+ 2684us) write_op.cc:260] PREPARE: finished
1026 14:59:17.733374 (+ 213us) write_op.cc:270] Start()
1026 14:59:17.733513 (+ 139us) write_op.cc:276] Timestamp: P: 1761490757733319 usec, L: 0
1026 14:59:17.736099 (+ 2586us) op_driver.cc:348] REPLICATION: starting
1026 14:59:17.879180 (+143081us) log.cc:844] Serialized 1893 byte log entry
1026 14:59:18.295422 (+416242us) op_driver.cc:464] REPLICATION: finished
1026 14:59:18.954315 (+658893us) write_op.cc:301] APPLY: starting
1026 14:59:18.963158 (+ 8843us) tablet.cc:1365] starting BulkCheckPresence
1026 14:59:18.981626 (+ 18468us) tablet.cc:1368] finished BulkCheckPresence
1026 14:59:18.981765 (+ 139us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 14:59:18.982328 (+ 563us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 14:59:18.982541 (+ 213us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
1026 14:59:18.982595 (+ 54us) write_op.cc:312] APPLY: finished
1026 14:59:18.982837 (+ 242us) log.cc:844] Serialized 40 byte log entry
1026 14:59:18.983028 (+ 191us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:18.983133 (+ 105us) write_op.cc:454] Released schema lock
1026 14:59:18.983178 (+ 45us) write_op.cc:341] FINISH: Updating metrics
1026 14:59:18.983580 (+ 402us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:18.983612 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3039047,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":626034,"apply.run_cpu_time_us":2389,"apply.run_wall_time_us":37020,"num_ops":4,"prepare.queue_time_us":248548,"prepare.run_cpu_time_us":21419,"prepare.run_wall_time_us":759748,"raft.queue_time_us":231261,"raft.run_cpu_time_us":41892,"raft.run_wall_time_us":584930,"replication_time_us":559117,"thread_start_us":1139055,"threads_started":4,"wal-append.queue_time_us":180420}]]}
I20251026 14:59:19.615250 24381 raft_consensus.cc:3055] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:19.629491 24381 raft_consensus.cc:740] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:19.591495 22725 rpcz_store.cc:276] 1026 14:59:17.460918 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:18.828772 (+1367854us) service_pool.cc:225] Handling call
1026 14:59:18.832930 (+ 4158us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:19.020251 (+187321us) raft_consensus.cc:1534] Early marking committed up to index 3
1026 14:59:19.020509 (+ 258us) log.cc:844] Serialized 14 byte log entry
1026 14:59:19.210295 (+189786us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:19.210376 (+ 81us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:19.210439 (+ 63us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:19.549599 (+339160us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b54003b3dd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:19.549647 (+ 48us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:19.549790 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5687,"prepare.queue_time_us":4892,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":68900,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":3657,"spinlock_wait_cycles":432625280,"thread_start_us":186127,"threads_started":1,"wal-append.queue_time_us":385045}
W20251026 14:59:19.563081 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:19.575098 24377 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:19.492657 24544 consensus_queue.cc:260] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.577627 24518 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:19.622490 22734 rpcz_store.cc:276] 1026 14:59:17.185961 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:18.164027 (+978066us) service_pool.cc:225] Handling call
1026 14:59:18.164412 (+ 385us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:18.202780 (+ 38368us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:18.202813 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:18.211413 (+ 8600us) log.cc:844] Serialized 52 byte log entry
1026 14:59:18.278172 (+ 66759us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:18.351005 (+ 72833us) log.cc:844] Serialized 14 byte log entry
1026 14:59:18.488773 (+137768us) spinlock_profiling.cc:243] Waited 20.3 ms on lock 0x7b180012cc40. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce026caa 00007fd8d7698a29 00007fd8cdee6c30 00007fd8ce0aea94 00007fd8ce0aea05 00007fd8d4b4db27 00007fd8d1168537 00007fd8d1177c4f 00007fd8d1177bc8
1026 14:59:18.489076 (+ 303us) log.cc:844] Serialized 14 byte log entry
1026 14:59:18.510188 (+ 21112us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:18.510317 (+ 129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:19.537763 (+1027446us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x7b2400390c38. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8cdf065a4 00007fd8cdf046ab 00007fd8cdf93409 00007fd8cdf93095 00007fd8cdf925f0 00007fd8d0d26498 00007fd8d0d26bf8 00007fd8d0cfe438 00007fd8d0cf981a 00007fd8d0cf8c66
1026 14:59:19.587063 (+ 49300us) raft_consensus.cc:1685] finished
1026 14:59:19.587095 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:19.587251 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":125154,"prepare.queue_time_us":238779,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":81,"raft.queue_time_us":59544,"raft.run_cpu_time_us":419,"raft.run_wall_time_us":13014,"spinlock_wait_cycles":51080192,"thread_start_us":50357,"threads_started":1,"wal-append.queue_time_us":601680}
I20251026 14:59:19.615859 24397 raft_consensus.cc:740] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
W20251026 14:59:19.660024 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 327464us
I20251026 14:59:19.570703 24405 consensus_queue.cc:1059] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.094s
I20251026 14:59:19.591106 24401 raft_consensus.cc:740] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:19.604759 24586 consensus_queue.cc:237] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
W20251026 14:59:19.669000 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:19.664723 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:19.665845 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:19.667539 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:19.626374 24228 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:19.671612 22658 raft_consensus.cc:1275] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251026 14:59:19.649291 24466 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:19.657859 24032 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:19.669871 24224 raft_consensus.cc:3060] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:19.658944 24518 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:19.682962 22773 leader_election.cc:400] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Vote denied by peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507) with higher term. Message: Invalid argument: T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6.
I20251026 14:59:19.649788 24354 raft_consensus.cc:2804] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:19.683974 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:19.684258 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:19.622874 24422 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.722018 22773 leader_election.cc:403] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20251026 14:59:19.676636 24544 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:19.662657 24579 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:19.655628 24582 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:19.686146 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:19.693184 22656 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e"
dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:46598
I20251026 14:59:19.693833 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:19.752753 22658 raft_consensus.cc:3060] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:19.694998 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.696524 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:19.696650 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.697232 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:19.697911 22651 raft_consensus.cc:1275] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:19.699718 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.755146 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.702402 22652 raft_consensus.cc:1240] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20251026 14:59:19.710435 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:19.757184 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:19.757910 22734 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:19.713762 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:19.667708 24103 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:19.682077 22650 raft_consensus.cc:1275] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20251026 14:59:19.722544 22804 raft_consensus.cc:2393] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:19.685328 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:19.657444 24377 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.749691 22799 raft_consensus.cc:2393] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:19.713197 24384 raft_consensus.cc:3055] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
W20251026 14:59:19.781729 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:19.684854 24310 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:19.810156 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.811138 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:19.811738 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 128915us
I20251026 14:59:19.819700 22728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1214) took 1728 ms. Trace:
I20251026 14:59:19.820127 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.826738 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:19.838096 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:19.718546 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:19.873379 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:19.942162 22656 raft_consensus.cc:606] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:19.942679 22806 raft_consensus.cc:2468] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:19.727298 24354 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:19.684131 24481 consensus_queue.cc:1048] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.125s
I20251026 14:59:19.882205 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.946619 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:19.885742 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.948096 22652 raft_consensus.cc:2468] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:19.719444 24594 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:19.920078 22658 raft_consensus.cc:2468] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:19.950345 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.808481 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:19.953420 22773 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:19.871800 24583 log.cc:927] Time spent T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.377s user 0.004s sys 0.000s
W20251026 14:59:19.961593 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 130060us
W20251026 14:59:19.923287 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 247410us
I20251026 14:59:19.930315 22734 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
W20251026 14:59:19.965792 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 154143us
I20251026 14:59:19.966995 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:19.967819 22734 raft_consensus.cc:2468] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:19.968255 23404 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:19.968688 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:19.930979 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:19.764931 22801 raft_consensus.cc:2393] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:19.969969 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:19.970763 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:19.971503 22801 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:19.972398 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:19.942301 22731 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:19.974067 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:19.975345 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:19.976008 22731 raft_consensus.cc:2468] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:19.976903 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:19.942196 22651 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:19.841464 24322 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:19.713563 24336 consensus_queue.cc:237] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.952564 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:19.940205 22728 rpcz_store.cc:276] 1026 14:59:18.091391 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:19.148756 (+1057365us) service_pool.cc:225] Handling call
1026 14:59:19.176769 (+ 28013us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:19.178167 (+ 1398us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:19.178201 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:19.181496 (+ 3295us) log.cc:844] Serialized 26 byte log entry
1026 14:59:19.369771 (+188275us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:19.369829 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:19.369928 (+ 99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:19.819479 (+449551us) raft_consensus.cc:1685] finished
1026 14:59:19.819500 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:19.819643 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2662,"raft.queue_time_us":1212,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":6187,"thread_start_us":187902,"threads_started":1,"wal-append.queue_time_us":345450}
I20251026 14:59:19.874161 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:19.977877 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.978343 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:19.911243 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:19.983076 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.750777 24397 consensus_queue.cc:260] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:19.838794 24590 consensus_queue.cc:1048] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.171s
I20251026 14:59:20.105669 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:19.985690 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:20.107225 22801 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:19.986465 22734 raft_consensus.cc:2393] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 4: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:19.810564 24579 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.080009 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:19.984298 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:20.110551 22804 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:19.985599 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:20.111841 22799 raft_consensus.cc:2410] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251026 14:59:20.111697 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:19.810415 24582 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.102996 22658 raft_consensus.cc:2368] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4.
I20251026 14:59:20.112702 22731 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:19.947556 24384 raft_consensus.cc:740] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:20.113137 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:20.126248 22733 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:20.113759 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:20.127044 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:20.118057 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:20.129571 22734 raft_consensus.cc:2468] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:19.979310 24466 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:20.130611 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:20.131500 22734 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:20.127089 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:19.954311 24354 raft_consensus.cc:3060] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:19.984424 24115 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:59:20.121181 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:20.142889 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:19.946702 24103 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:20.120424 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 54622.6us
I20251026 14:59:20.123471 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:20.153653 22659 raft_consensus.cc:2393] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 4: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:19.809346 24401 consensus_queue.cc:260] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:20.149111 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 180117us
W20251026 14:59:20.162088 24592 log.cc:927] Time spent T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.399s user 0.006s sys 0.003s
I20251026 14:59:20.128484 24310 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:20.127779 24397 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:20.180472 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:20.104439 24536 consensus_queue.cc:1059] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.062s
I20251026 14:59:20.165571 24602 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:20.274130 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:19.987455 24594 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.281610 23401 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:20.117837 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:20.140576 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:20.144352 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 163677us
W20251026 14:59:20.114601 24345 consensus_meta.cc:287] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.562s user 0.006s sys 0.003s
I20251026 14:59:19.952651 24554 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:19.990513 24593 raft_consensus.cc:2804] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:20.149199 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:20.149709 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:19.992105 22802 raft_consensus.cc:2468] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:59:20.155884 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:20.128154 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:20.165639 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:20.167500 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:20.171298 22651 raft_consensus.cc:2410] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251026 14:59:20.155279 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:20.132503 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:20.176644 22774 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:19.991266 24228 consensus_queue.cc:1048] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.160s
I20251026 14:59:20.164983 24605 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:20.188442 22653 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
W20251026 14:59:20.190001 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:20.205878 22726 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.001s sys 0.002s
W20251026 14:59:20.321794 22726 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.700s user 0.001s sys 0.003s
I20251026 14:59:20.322781 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1213) took 2231 ms. Trace:
W20251026 14:59:20.206306 24385 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.003s sys 0.000s
I20251026 14:59:20.199546 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:20.183295 24310 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.132422 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:20.133531 22733 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
W20251026 14:59:20.270452 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 260401us
W20251026 14:59:20.276155 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:20.328430 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:20.328861 24142 raft_consensus.cc:2749] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:20.136147 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:59:20.187656 24584 log.cc:927] Time spent T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.514s user 0.000s sys 0.001s
I20251026 14:59:20.179024 24587 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:20.146839 24466 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.350324 24532 consensus_queue.cc:1059] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:59:20.350780 24466 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:20.318957 24481 consensus_queue.cc:1048] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20251026 14:59:20.351495 24466 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:20.351956 24466 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.290436 24377 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:20.154397 24440 consensus_queue.cc:1048] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.120s
I20251026 14:59:20.306183 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:20.328118 24385 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.794s user 0.004s sys 0.000s
I20251026 14:59:20.311107 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:20.313459 22731 raft_consensus.cc:2393] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 2: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:20.275866 24309 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:20.369302 24309 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.371778 24309 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:20.316042 22659 raft_consensus.cc:2393] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 2: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:20.323637 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:20.301885 24345 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.374450 22728 raft_consensus.cc:1240] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251026 14:59:20.375576 24345 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:20.375957 22728 raft_consensus.cc:1240] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: [2.2-5.3]
I20251026 14:59:20.376742 24631 consensus_queue.cc:1059] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
W20251026 14:59:20.377878 24609 log.cc:927] Time spent T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20251026 14:59:20.381817 24582 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:20.382174 22731 raft_consensus.cc:1240] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:20.323016 24597 raft_consensus.cc:3055] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:20.319257 22804 raft_consensus.cc:2393] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:20.315447 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:20.322023 24610 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:20.393232 22651 raft_consensus.cc:1275] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20251026 14:59:20.243184 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:20.323232 22726 rpcz_store.cc:276] 1026 14:59:18.091049 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:19.143265 (+1052216us) service_pool.cc:225] Handling call
1026 14:59:19.143610 (+ 345us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:19.143977 (+ 367us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:19.144006 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:19.152478 (+ 8472us) log.cc:844] Serialized 52 byte log entry
1026 14:59:19.495397 (+342919us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:19.621233 (+125836us) log.cc:844] Serialized 14 byte log entry
1026 14:59:20.322184 (+700951us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:20.322464 (+ 280us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:20.322524 (+ 60us) raft_consensus.cc:1685] finished
1026 14:59:20.322548 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:20.322707 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":111224,"prepare.queue_time_us":96310,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":350752,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":2735,"thread_start_us":1003967,"threads_started":2,"wal-append.queue_time_us":484343}
I20251026 14:59:20.334820 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:20.312381 24606 log.cc:927] Time spent T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.167s user 0.001s sys 0.000s
W20251026 14:59:20.342758 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 166238us
W20251026 14:59:20.341918 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 55679.4us
I20251026 14:59:20.336635 22803 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:20.337932 24587 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.346698 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:20.400089 22774 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:20.307745 24593 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:20.283103 24594 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:20.321389 24479 consensus_meta.cc:287] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.942s user 0.003s sys 0.008s
I20251026 14:59:20.299052 22801 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:20.362008 22800 raft_consensus.cc:2393] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 4: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
W20251026 14:59:20.368685 24601 log.cc:927] Time spent T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.264s user 0.000s sys 0.001s
I20251026 14:59:20.316830 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:20.195180 24384 consensus_queue.cc:260] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.348526 24310 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:20.402223 24455 raft_consensus.cc:3055] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:20.403079 22804 raft_consensus.cc:1240] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
W20251026 14:59:20.403666 22774 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.2.1:46185 timed out after 1.592s (SENT) [suppressed 7 similar messages]
W20251026 14:59:20.404376 24308 consensus_meta.cc:287] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.943s user 0.005s sys 0.005s
I20251026 14:59:20.405799 22805 raft_consensus.cc:1240] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:20.334513 24422 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:20.380389 24478 consensus_meta.cc:287] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.772s user 0.007s sys 0.000s
I20251026 14:59:20.422329 22728 raft_consensus.cc:1240] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
I20251026 14:59:20.419027 22801 raft_consensus.cc:1240] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:20.421336 22622 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:20.466574 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 45348.7us
I20251026 14:59:20.425504 22731 raft_consensus.cc:1275] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20251026 14:59:20.415469 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:20.432741 22734 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 6
W20251026 14:59:20.339535 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 178501us
I20251026 14:59:20.437395 22803 raft_consensus.cc:1240] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:20.313545 24554 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.414826 24593 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:20.391916 24597 raft_consensus.cc:740] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:20.407259 24031 raft_consensus.cc:3055] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:20.452822 22657 raft_consensus.cc:1240] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251026 14:59:20.436722 24403 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:20.283382 24602 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.461848 22729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1208) took 2372 ms. Trace:
I20251026 14:59:20.505846 22734 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:20.505486 22729 rpcz_store.cc:276] 1026 14:59:18.089590 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:19.131846 (+1042256us) service_pool.cc:225] Handling call
1026 14:59:19.135314 (+ 3468us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:19.480687 (+345373us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:19.480741 (+ 54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:19.486270 (+ 5529us) log.cc:844] Serialized 26 byte log entry
1026 14:59:19.739430 (+253160us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:19.739495 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:19.739635 (+ 140us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:20.461603 (+721968us) raft_consensus.cc:1685] finished
1026 14:59:20.461632 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:20.461777 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8418,"raft.queue_time_us":26637,"raft.run_cpu_time_us":67,"raft.run_wall_time_us":67,"thread_start_us":561955,"threads_started":2,"wal-append.queue_time_us":495543}
I20251026 14:59:20.473397 22622 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:20.474913 24384 raft_consensus.cc:3060] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:20.499488 24031 raft_consensus.cc:740] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:20.514951 24507 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:20.515602 24031 consensus_queue.cc:260] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.516767 24031 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:20.330430 24605 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.518580 24587 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:20.518790 24605 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:20.499480 24476 consensus_meta.cc:287] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.981s user 0.010s sys 0.001s
I20251026 14:59:20.481602 24385 raft_consensus.cc:2804] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:20.520615 24385 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:20.520728 24629 raft_consensus.cc:3055] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:20.390383 24377 consensus_queue.cc:1048] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.063s
I20251026 14:59:20.507735 24403 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.521852 24381 consensus_queue.cc:260] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.522902 24381 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:20.523423 24403 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:20.396536 24613 consensus_queue.cc:1048] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
W20251026 14:59:20.524325 24621 log.cc:927] Time spent T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.156s user 0.000s sys 0.000s
W20251026 14:59:20.526563 24518 consensus_meta.cc:287] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.718s user 0.009s sys 0.000s
I20251026 14:59:20.526777 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:20.527006 24518 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.415156 24610 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:20.436594 24536 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:20.529668 24610 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:20.529543 24536 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.492053 24466 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:20.451948 24625 raft_consensus.cc:2804] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:20.504648 22799 raft_consensus.cc:1240] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
W20251026 14:59:20.450184 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:20.508241 22806 raft_consensus.cc:3060] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:20.475777 24308 raft_consensus.cc:2749] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:20.431922 24418 consensus_queue.cc:1059] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20251026 14:59:20.509881 24440 raft_consensus.cc:2804] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:20.488711 24630 raft_consensus.cc:2749] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:20.513834 24103 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:20.482084 24478 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.514050 24597 consensus_queue.cc:260] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.431924 24479 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:20.519888 24476 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.447187 24455 raft_consensus.cc:740] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:20.521013 24385 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
W20251026 14:59:20.503357 24333 consensus_meta.cc:287] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 1.005s user 0.006s sys 0.005s
I20251026 14:59:20.512858 24554 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:20.536360 24463 consensus_meta.cc:287] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 1.668s user 0.010s sys 0.004s
I20251026 14:59:20.521106 24629 raft_consensus.cc:740] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:20.531510 24536 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:20.538743 24353 consensus_meta.cc:287] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 1.245s user 0.010s sys 0.001s
I20251026 14:59:20.535733 24625 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:20.538272 24579 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:20.582123 22730 raft_consensus.cc:1240] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: []
W20251026 14:59:20.575026 24578 consensus_meta.cc:287] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 1.146s user 0.009s sys 0.004s
I20251026 14:59:20.529035 24518 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:20.586138 22697 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:20.520128 24032 consensus_meta.cc:287] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.802s user 0.002s sys 0.005s
I20251026 14:59:20.567118 24440 raft_consensus.cc:697] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:20.572326 24153 raft_consensus.cc:3055] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:20.627010 24153 raft_consensus.cc:740] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:20.625746 22700 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:20.627871 24153 consensus_queue.cc:260] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.605755 24455 consensus_queue.cc:260] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.587538 24625 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:20.629467 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1212) took 2538 ms. Trace:
I20251026 14:59:20.629927 22732 rpcz_store.cc:276] 1026 14:59:18.090761 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:19.137390 (+1046629us) service_pool.cc:225] Handling call
1026 14:59:19.144723 (+ 7333us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:19.435659 (+290936us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:19.435694 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:19.458195 (+ 22501us) log.cc:844] Serialized 52 byte log entry
1026 14:59:19.886812 (+428617us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:19.886876 (+ 64us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:19.886973 (+ 97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:20.629207 (+742234us) raft_consensus.cc:1685] finished
1026 14:59:20.629228 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:20.629410 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":17416,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":84,"thread_start_us":658113,"threads_started":2,"wal-append.queue_time_us":1134686}
I20251026 14:59:20.631762 22700 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:20.610209 24602 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:20.611606 24478 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:20.614055 22727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1217) took 2354 ms. Trace:
W20251026 14:59:20.634864 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 48846.7us
I20251026 14:59:20.615474 24479 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:20.582746 24597 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:20.617427 24322 consensus_meta.cc:287] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.525s user 0.010s sys 0.000s
I20251026 14:59:20.618505 24354 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:20.618319 24629 consensus_queue.cc:260] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:59:20.621466 24103 consensus_queue.cc:1059] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 6.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:20.625478 24440 consensus_queue.cc:237] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.624197 24476 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:20.587347 24353 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.607786 24401 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
W20251026 14:59:20.609140 24544 consensus_meta.cc:287] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.847s user 0.010s sys 0.000s
I20251026 14:59:20.629848 24455 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:20.641322 24308 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:20.682830 22807 raft_consensus.cc:2410] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251026 14:59:20.573750 24333 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.577211 24463 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:20.644327 24611 log.cc:927] Time spent T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.210s user 0.000s sys 0.002s
I20251026 14:59:20.647495 22697 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:20.686856 22697 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:20.687196 24463 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:20.687984 22801 raft_consensus.cc:1240] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:20.688879 22697 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:20.690104 24476 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:20.646947 22727 rpcz_store.cc:276] 1026 14:59:18.259784 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:19.232821 (+973037us) service_pool.cc:225] Handling call
1026 14:59:19.233195 (+ 374us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:19.281183 (+ 47988us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:19.281232 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:19.284294 (+ 3062us) log.cc:844] Serialized 26 byte log entry
1026 14:59:19.637245 (+352951us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:19.637544 (+ 299us) log.cc:844] Serialized 14 byte log entry
1026 14:59:19.637706 (+ 162us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:19.637776 (+ 70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:20.613820 (+976044us) raft_consensus.cc:1685] finished
1026 14:59:20.613847 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:20.613990 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":71616,"prepare.queue_time_us":11553,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":510952,"raft.run_cpu_time_us":409,"raft.run_wall_time_us":42933,"thread_start_us":352650,"threads_started":1,"wal-append.queue_time_us":1303605}
I20251026 14:59:20.690443 24640 raft_consensus.cc:2749] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:20.690646 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:20.691062 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:20.651391 22654 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
W20251026 14:59:20.622599 24224 consensus_meta.cc:287] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.902s user 0.011s sys 0.000s
I20251026 14:59:20.629166 24153 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:20.603094 24578 raft_consensus.cc:2749] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:20.692337 22727 raft_consensus.cc:2468] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:20.659416 22806 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:20.669658 22700 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:20.693642 22727 raft_consensus.cc:1240] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:20.694478 22700 leader_election.cc:400] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6.
I20251026 14:59:20.694790 22700 leader_election.cc:403] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20251026 14:59:20.691515 22697 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:20.695305 24224 raft_consensus.cc:2804] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader election won for term 2
I20251026 14:59:20.695730 22700 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:20.696365 24640 raft_consensus.cc:2749] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6.
I20251026 14:59:20.697492 22700 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:20.698187 22697 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:20.698493 24479 raft_consensus.cc:2804] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:20.699008 24479 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:20.699570 24479 raft_consensus.cc:3060] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:20.699651 24640 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:20.678329 24353 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:20.646176 24579 raft_consensus.cc:2804] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:20.684696 22652 raft_consensus.cc:1240] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20251026 14:59:20.684751 22658 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:20.635020 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:20.704414 22624 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:20.705058 22732 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:20.695818 24224 raft_consensus.cc:697] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 2 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:20.699038 24602 raft_consensus.cc:2804] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:20.652783 24610 raft_consensus.cc:2804] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:20.702091 22697 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:20.707437 22624 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:20.702123 22773 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:20.689520 24626 raft_consensus.cc:2804] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:20.711148 22725 raft_consensus.cc:1240] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20251026 14:59:20.672801 24629 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:20.715610 24333 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:20.709740 22773 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:20.717672 22773 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:20.703254 24579 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:20.654286 24322 raft_consensus.cc:2749] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507) with higher term. Message: Invalid argument: T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6.
I20251026 14:59:20.712350 24626 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:20.712498 24602 raft_consensus.cc:697] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:20.712535 22732 raft_consensus.cc:1275] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20251026 14:59:20.727036 22624 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:20.728242 22773 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:20.672942 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:20.791617 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:20.833344 22653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1213) took 1110 ms. Trace:
I20251026 14:59:20.834517 22653 rpcz_store.cc:276] 1026 14:59:19.723103 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:19.810891 (+ 87788us) service_pool.cc:225] Handling call
1026 14:59:20.833147 (+1022256us) spinlock_profiling.cc:243] Waited 248 ms on lock 0x7b54001d79d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:20.833255 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":571172224}
W20251026 14:59:20.837118 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 108974us
I20251026 14:59:20.826010 22807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44594 (request call id 971) took 1238 ms. Trace:
I20251026 14:59:20.792956 22697 leader_election.cc:304] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:20.837929 22773 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:20.836323 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 432644us
I20251026 14:59:20.711306 24463 raft_consensus.cc:2749] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251026 14:59:20.838753 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 45935.8us
W20251026 14:59:20.833252 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:20.844307 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:20.844791 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:20.837811 22807 rpcz_store.cc:276] 1026 14:59:19.587976 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:19.588108 (+ 132us) service_pool.cc:225] Handling call
1026 14:59:20.825693 (+1237585us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b540020b7d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:20.825921 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2472237824}
I20251026 14:59:20.845580 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:20.846215 22697 leader_election.cc:304] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
W20251026 14:59:20.839818 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 112862us
I20251026 14:59:20.847088 22624 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:20.843120 22773 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:20.724680 24578 raft_consensus.cc:2804] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:20.800971 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1208) took 1092 ms. Trace:
W20251026 14:59:20.859869 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:20.723182 24353 raft_consensus.cc:2764] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20251026 14:59:20.792032 24579 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:20.791330 24610 raft_consensus.cc:697] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:20.869565 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:20.872629 22773 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:20.791503 24405 raft_consensus.cc:2804] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:20.867767 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:20.835556 24032 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:20.870993 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:20.851269 24638 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
W20251026 14:59:20.834332 22542 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 64248.8us
I20251026 14:59:20.840575 24333 raft_consensus.cc:2749] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:20.846663 24322 raft_consensus.cc:2804] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:20.836014 24626 raft_consensus.cc:3060] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:20.871001 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:20.882831 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:20.965628 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:20.966025 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:20.966274 22808 raft_consensus.cc:2468] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:20.791060 24544 raft_consensus.cc:2804] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:20.844060 24634 raft_consensus.cc:2804] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:20.850055 24578 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:20.850904 24527 raft_consensus.cc:2804] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:20.916793 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:20.968007 22650 raft_consensus.cc:2410] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:20.852649 24507 raft_consensus.cc:2749] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:20.871135 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:20.863969 24635 raft_consensus.cc:2749] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251026 14:59:20.909776 24652 log.cc:927] Time spent T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20251026 14:59:20.869226 24589 raft_consensus.cc:2749] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:20.873245 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:20.967880 22726 raft_consensus.cc:2468] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:21.030333 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:21.031440 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:21.032027 22726 raft_consensus.cc:2468] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:20.913887 24405 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:21.033313 22773 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:21.033768 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:20.969226 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:21.018121 22730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1218) took 2182 ms. Trace:
I20251026 14:59:21.035856 22730 rpcz_store.cc:276] 1026 14:59:18.835487 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:19.257855 (+422368us) service_pool.cc:225] Handling call
1026 14:59:19.283107 (+ 25252us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:20.588321 (+1305214us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:21.017785 (+429464us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54002dc654. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:21.018056 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2986894720}
I20251026 14:59:20.928702 24308 raft_consensus.cc:2749] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251026 14:59:20.930563 24492 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.412s user 0.004s sys 0.042s
I20251026 14:59:20.970184 24527 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:21.038009 22657 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:21.041407 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:20.926386 24628 raft_consensus.cc:2804] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:20.963172 24310 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:21.018291 24115 raft_consensus.cc:2804] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:20.964735 24322 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:20.926870 24032 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.928639 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:21.022085 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:21.071421 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:21.045574 24492 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.552s user 0.005s sys 0.043s
I20251026 14:59:21.059377 22622 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:21.039479 24507 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:21.056102 24527 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:21.019068 24593 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.970047 24634 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:21.046201 24354 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:21.086150 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:21.087044 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:21.036288 24405 raft_consensus.cc:3060] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:20.929212 24638 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.075250 22622 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:21.070847 24630 raft_consensus.cc:2804] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:21.078455 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:20.928289 24224 consensus_queue.cc:237] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.055182 24628 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:21.083379 22655 raft_consensus.cc:1240] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:21.066632 24322 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:21.098446 22622 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:21.111690 22657 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:21.131510 22658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1202) took 1427 ms. Trace:
I20251026 14:59:21.080940 24153 raft_consensus.cc:2749] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:20.969631 24578 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:21.070588 24310 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.029779 24602 consensus_queue.cc:237] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.137634 22807 raft_consensus.cc:2393] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:21.055490 24610 consensus_queue.cc:237] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.129448 22801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 999) took 1155 ms. Trace:
I20251026 14:59:21.111054 24628 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:21.129916 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:21.131712 24336 raft_consensus.cc:2749] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:21.118753 24638 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:21.134099 24639 log.cc:927] Time spent T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.556s user 0.002s sys 0.000s
W20251026 14:59:21.126782 24655 log.cc:927] Time spent T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20251026 14:59:21.124625 22805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 982) took 1403 ms. Trace:
I20251026 14:59:21.093181 24634 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:21.134487 24532 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:21.172001 22700 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:21.228655 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 56745.3us
I20251026 14:59:21.172622 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:21.235807 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:21.237336 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:21.228017 24638 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:21.173121 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:59:21.121765 22808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1025) took 1437 ms (client timeout 1683 ms). Trace:
W20251026 14:59:21.247905 22808 rpcz_store.cc:269] 1026 14:59:19.683919 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:19.685052 (+ 1133us) service_pool.cc:225] Handling call
1026 14:59:21.121598 (+1436546us) spinlock_profiling.cc:243] Waited 900 ms on lock 0x7b540003a7d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:21.121706 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2070670208}
I20251026 14:59:21.252229 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:21.133030 23401 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:21.238742 24354 raft_consensus.cc:2804] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election won for term 6
W20251026 14:59:21.173497 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:21.084568 24381 raft_consensus.cc:493] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:21.106426 22730 raft_consensus.cc:2393] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 6: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:21.081169 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:21.118240 24418 raft_consensus.cc:3055] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:21.111795 24463 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:21.178167 24640 raft_consensus.cc:2749] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:21.101418 24492 raft_consensus.cc:2749] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251026 14:59:21.184691 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 151472us
I20251026 14:59:21.185000 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:21.269898 22653 raft_consensus.cc:2468] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:21.270819 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:21.271569 22653 raft_consensus.cc:2393] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:21.272359 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:21.272716 22773 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:21.131561 24632 log.cc:927] Time spent T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.562s user 0.000s sys 0.002s
I20251026 14:59:21.186121 22658 rpcz_store.cc:276] 1026 14:59:19.704106 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:20.131500 (+427394us) service_pool.cc:225] Handling call
1026 14:59:21.131322 (+999822us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x7b54000dc854. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:21.131450 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":785410304}
I20251026 14:59:21.193231 22801 rpcz_store.cc:276] 1026 14:59:19.974638 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:20.465598 (+490960us) service_pool.cc:225] Handling call
1026 14:59:20.465920 (+ 322us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:20.688413 (+222493us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:21.129242 (+440829us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b5400333bd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:21.129400 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":510227712}
I20251026 14:59:21.111500 24630 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:21.215813 22800 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:21.274724 24463 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:21.223836 22697 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:21.225555 22805 rpcz_store.cc:276] 1026 14:59:19.721254 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:20.354517 (+633263us) service_pool.cc:225] Handling call
1026 14:59:20.395234 (+ 40717us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:20.463395 (+ 68161us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:21.124428 (+661033us) spinlock_profiling.cc:243] Waited 40.3 ms on lock 0x7b54001bf150. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:21.124565 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":92745984}
W20251026 14:59:21.283921 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 60212us
I20251026 14:59:21.283998 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:21.285068 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:21.074297 24544 raft_consensus.cc:697] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:21.238071 22650 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:21.249603 24638 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.253383 22808 raft_consensus.cc:2468] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:21.231184 24153 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:21.298844 22734 raft_consensus.cc:1275] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:21.299103 24153 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.300834 22734 raft_consensus.cc:1240] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
I20251026 14:59:21.228296 24532 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.133738 24411 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:21.263582 22774 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:21.276459 24455 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:21.274531 22731 raft_consensus.cc:1275] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251026 14:59:21.275580 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:21.279493 22728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 841) took 1366 ms. Trace:
I20251026 14:59:21.277822 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:21.280606 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:21.346712 22800 raft_consensus.cc:3060] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:59:21.267428 24354 raft_consensus.cc:697] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:21.306604 22731 raft_consensus.cc:1240] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:21.308190 22733 raft_consensus.cc:1240] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20251026 14:59:21.285761 22805 raft_consensus.cc:2393] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:21.312012 22728 rpcz_store.cc:276] 1026 14:59:19.913462 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:20.463551 (+550089us) service_pool.cc:225] Handling call
1026 14:59:20.500234 (+ 36683us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:20.500415 (+ 181us) raft_consensus.cc:1534] Early marking committed up to index 3
1026 14:59:20.836612 (+336197us) log.cc:844] Serialized 14 byte log entry
1026 14:59:21.279134 (+442522us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:21.279185 (+ 51us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:21.279223 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:21.279313 (+ 90us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:21.279417 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":118470,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":83,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":7138,"spinlock_wait_cycles":17664,"thread_start_us":764091,"threads_started":2}
I20251026 14:59:21.286707 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1206) took 1578 ms. Trace:
I20251026 14:59:21.352574 22659 rpcz_store.cc:276] 1026 14:59:19.707939 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:20.372784 (+664845us) service_pool.cc:225] Handling call
1026 14:59:20.373154 (+ 370us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:20.405769 (+ 32615us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:20.405801 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:20.406391 (+ 590us) log.cc:844] Serialized 52 byte log entry
1026 14:59:20.837049 (+430658us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:21.263740 (+426691us) log.cc:844] Serialized 14 byte log entry
1026 14:59:21.286390 (+ 22650us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:21.286469 (+ 79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:21.286511 (+ 42us) raft_consensus.cc:1685] finished
1026 14:59:21.286523 (+ 12us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:21.286646 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46446,"prepare.queue_time_us":34784,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":44393,"raft.run_cpu_time_us":274,"raft.run_wall_time_us":20160,"thread_start_us":762262,"threads_started":2,"wal-append.queue_time_us":386803}
I20251026 14:59:21.354548 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:21.355234 22659 raft_consensus.cc:2393] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 5: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:21.083747 24507 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.368557 24507 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:21.372098 24634 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.374071 22733 raft_consensus.cc:1240] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20251026 14:59:21.374364 24634 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:21.374909 24634 raft_consensus.cc:2749] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:21.289630 22700 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:21.290270 22697 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:21.266328 24418 raft_consensus.cc:740] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:59:21.375627 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 86051.9us
I20251026 14:59:21.376102 24418 consensus_queue.cc:260] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.2, Last appended by leader: 2, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.377080 24418 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:21.267035 24664 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:21.263793 24381 raft_consensus.cc:515] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:21.377813 24664 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.378849 22658 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:21.379544 24381 leader_election.cc:290] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:21.379853 24664 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:21.287722 24384 raft_consensus.cc:2749] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:21.294673 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:21.291285 24032 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:21.316556 24405 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:20.848686 22651 rpcz_store.cc:276] 1026 14:59:19.708612 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:20.394075 (+685463us) service_pool.cc:225] Handling call
1026 14:59:20.394370 (+ 295us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:20.394562 (+ 192us) raft_consensus.cc:1534] Early marking committed up to index 4
1026 14:59:20.507177 (+112615us) log.cc:844] Serialized 14 byte log entry
1026 14:59:20.800013 (+292836us) log.cc:844] Serialized 14 byte log entry
1026 14:59:20.800401 (+ 388us) log.cc:844] Serialized 14 byte log entry
1026 14:59:20.800556 (+ 155us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:20.800600 (+ 44us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:20.800638 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:20.800727 (+ 89us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:20.800896 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49163,"prepare.queue_time_us":166480,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":149028,"spinlock_wait_cycles":23835008,"thread_start_us":258139,"threads_started":2,"wal-append.queue_time_us":182523}
I20251026 14:59:21.387950 22728 raft_consensus.cc:1240] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251026 14:59:21.291380 24332 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:21.282658 24630 raft_consensus.cc:3060] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:21.297938 24593 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:21.291667 24640 raft_consensus.cc:2749] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20251026 14:59:21.264496 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 406461us
I20251026 14:59:21.402930 22624 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:21.304508 22656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:46598 (request call id 2) took 1612 ms. Trace:
I20251026 14:59:21.323741 22806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44594 (request call id 988) took 1361 ms. Trace:
I20251026 14:59:21.254637 22657 raft_consensus.cc:2410] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251026 14:59:21.348089 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:21.333535 22655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1204) took 1626 ms. Trace:
I20251026 14:59:21.307756 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1230) took 1594 ms. Trace:
I20251026 14:59:21.340489 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:21.324046 22802 raft_consensus.cc:1240] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5. Ops: []
W20251026 14:59:21.345604 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:21.301262 24153 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:21.409376 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:21.345748 22653 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:21.357134 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:21.410681 22808 raft_consensus.cc:2393] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:21.280654 24554 raft_consensus.cc:2804] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251026 14:59:21.288293 24479 consensus_meta.cc:287] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.588s user 0.009s sys 0.005s
I20251026 14:59:21.293794 24310 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:21.332051 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:21.379664 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 89441.1us
I20251026 14:59:21.432588 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:21.381410 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:21.433959 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:21.433966 24379 consensus_queue.cc:1059] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251026 14:59:21.434242 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:21.435711 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:21.297528 24463 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:21.388746 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:21.298841 24657 log.cc:927] Time spent T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.124s user 0.001s sys 0.000s
I20251026 14:59:21.390564 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:21.316854 24401 consensus_meta.cc:287] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.633s user 0.006s sys 0.005s
W20251026 14:59:21.336239 24661 log.cc:927] Time spent T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.109s user 0.001s sys 0.000s
W20251026 14:59:21.328236 24656 log.cc:927] Time spent T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
W20251026 14:59:21.348522 24646 log.cc:927] Time spent T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
W20251026 14:59:21.293830 24599 log.cc:927] Time spent T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 1.413s user 0.015s sys 0.007s
W20251026 14:59:21.299103 24476 consensus_meta.cc:287] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.609s user 0.004s sys 0.004s
I20251026 14:59:21.404455 22806 rpcz_store.cc:276] 1026 14:59:19.962527 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:20.112371 (+149844us) service_pool.cc:225] Handling call
1026 14:59:21.323512 (+1211141us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b540038aed4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:21.323675 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":447131392}
I20251026 14:59:21.426855 24479 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.405647 22655 rpcz_store.cc:276] 1026 14:59:19.707302 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:20.337900 (+630598us) service_pool.cc:225] Handling call
1026 14:59:20.338284 (+ 384us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:21.110800 (+772516us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:21.333239 (+222439us) spinlock_profiling.cc:243] Waited 745 ms on lock 0x7b540012f2d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:21.333471 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1712923392}
I20251026 14:59:21.405714 22725 rpcz_store.cc:276] 1026 14:59:19.720604 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:19.739925 (+ 19321us) service_pool.cc:225] Handling call
1026 14:59:19.750393 (+ 10468us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:20.711704 (+961311us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:21.307509 (+595805us) spinlock_profiling.cc:243] Waited 960 ms on lock 0x7b54000b9ad4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:21.307698 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2208821632}
I20251026 14:59:21.291226 24353 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:21.406162 22807 raft_consensus.cc:2393] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:21.404336 22656 rpcz_store.cc:276] 1026 14:59:19.692363 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:19.692987 (+ 624us) service_pool.cc:225] Handling call
1026 14:59:21.304351 (+1611364us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b54002a85d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d11583f3 00007fd8d6109bca 00007fd8cf6b2934 00007fd8cf6b28c2 00007fd8cf6b27f1 00007fd8cf6b276c 00007fd8cf6b1a22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:21.304430 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":436440832}
I20251026 14:59:21.410925 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:21.411645 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:21.460414 24629 consensus_meta.cc:287] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.738s user 0.014s sys 0.000s
I20251026 14:59:21.414650 22729 raft_consensus.cc:1240] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20251026 14:59:21.483702 22732 raft_consensus.cc:1240] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:21.302978 24532 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:21.305672 24455 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.426163 22800 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:59:21.415119 24554 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:21.396715 24635 raft_consensus.cc:2804] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:21.428593 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:21.475796 24385 consensus_meta.cc:287] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.900s user 0.009s sys 0.001s
I20251026 14:59:21.431072 22653 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:21.429450 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:21.441239 22624 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:21.442049 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:21.303527 24411 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.316262 24527 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.440505 22651 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:21.337949 24631 consensus_queue.cc:1059] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.086s
W20251026 14:59:21.415282 24597 consensus_meta.cc:287] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.761s user 0.008s sys 0.002s
I20251026 14:59:21.447679 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:21.408001 24032 raft_consensus.cc:2749] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:21.423774 22553 catalog_manager.cc:5649] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 2 to 4, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 4 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } }
W20251026 14:59:21.536015 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 97522.5us
I20251026 14:59:21.423535 23401 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
W20251026 14:59:21.459100 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:21.461019 22799 raft_consensus.cc:1275] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20251026 14:59:21.400660 24332 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:21.462303 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:21.338582 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1199) took 1635 ms. Trace:
W20251026 14:59:21.464836 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 163359us
I20251026 14:59:21.541744 22806 raft_consensus.cc:2410] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 8 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
W20251026 14:59:21.541940 22654 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1195) took 1845 ms (client timeout 2200 ms). Trace:
W20251026 14:59:21.542338 22654 rpcz_store.cc:269] 1026 14:59:19.696623 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:19.697005 (+ 382us) service_pool.cc:225] Handling call
1026 14:59:21.541772 (+1844767us) spinlock_profiling.cc:243] Waited 880 ms on lock 0x7b540010e0d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:21.541886 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2023220608}
I20251026 14:59:21.466792 22803 raft_consensus.cc:2393] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:21.337299 24115 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:21.466014 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:21.448928 24103 consensus_queue.cc:1048] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.030s
I20251026 14:59:21.545917 24638 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:21.453558 24544 consensus_queue.cc:237] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.465340 24354 consensus_queue.cc:237] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:21.547329 24625 consensus_meta.cc:287] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.913s user 0.009s sys 0.005s
I20251026 14:59:21.547760 24625 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.469159 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:21.549582 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:21.550202 22803 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:21.551069 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:21.551321 22656 raft_consensus.cc:1240] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20251026 14:59:21.552070 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:21.552681 24115 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:21.531560 24385 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.474115 24353 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.483323 24479 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:21.476910 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:21.460336 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:21.500746 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:21.507208 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 243719us
I20251026 14:59:21.507675 22800 raft_consensus.cc:1275] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:21.463848 24476 raft_consensus.cc:2749] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:21.391408 24518 raft_consensus.cc:2749] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20251026 14:59:21.473738 24607 log.cc:927] Time spent T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.932s user 0.000s sys 0.009s
W20251026 14:59:21.528339 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 87180.2us
I20251026 14:59:21.573884 22624 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:21.574662 24628 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.575778 22805 raft_consensus.cc:1240] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:21.576402 24403 raft_consensus.cc:2804] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:21.576586 24411 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:21.576882 24403 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:21.577564 24403 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:21.531441 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:21.578644 24476 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:21.533088 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:21.579145 24476 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.579327 24322 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.580165 22651 raft_consensus.cc:1240] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:21.533427 22700 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:21.527527 24626 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:21.581215 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 47869.6us
I20251026 14:59:21.582867 24626 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:21.583422 24626 raft_consensus.cc:2804] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:21.583747 24626 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:21.584090 24626 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:21.584581 22727 raft_consensus.cc:1240] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:21.585682 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:21.586333 24527 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:21.536653 22553 catalog_manager.cc:5649] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 2 to 3, leader changed from df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 3 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:21.538017 22801 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:21.538239 22799 pending_rounds.cc:85] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821: Aborting all ops after (but not including) 2
I20251026 14:59:21.509024 24554 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:21.527220 24635 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:21.541381 22652 rpcz_store.cc:276] 1026 14:59:19.703180 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:20.326821 (+623641us) service_pool.cc:225] Handling call
1026 14:59:20.357163 (+ 30342us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:20.705294 (+348131us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:21.338297 (+633003us) spinlock_profiling.cc:243] Waited 327 ms on lock 0x7b540009ddd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:21.338516 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":751014144}
I20251026 14:59:21.552811 22803 raft_consensus.cc:2410] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251026 14:59:21.532284 24440 raft_consensus.cc:2804] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:21.554164 22656 raft_consensus.cc:1240] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: [3.3-5.4]
I20251026 14:59:21.560779 22807 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:21.562335 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:21.562335 22774 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:21.593261 22806 raft_consensus.cc:2468] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 1.
I20251026 14:59:21.594240 22774 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:21.556118 24385 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:21.564461 22622 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:21.566689 22659 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
W20251026 14:59:21.571342 24579 consensus_meta.cc:287] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.665s user 0.009s sys 0.000s
I20251026 14:59:21.576865 24031 consensus_queue.cc:1048] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:59:21.585501 24103 consensus_queue.cc:1059] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:59:21.585525 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:21.587630 22799 pending_rounds.cc:107] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20251026 14:59:21.588038 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:21.591567 24479 consensus_queue.cc:1059] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20251026 14:59:21.530305 22650 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:21.555253 24634 raft_consensus.cc:3055] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:21.589399 22654 raft_consensus.cc:1275] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:21.590819 22553 catalog_manager.cc:5649] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:21.563935 22800 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522"
dest_uuid: "0d684445287b450a8f738b6ad4b59821"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:44012
I20251026 14:59:21.600577 22774 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:21.601883 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:21.692179 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 75155.2us
W20251026 14:59:21.695031 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 94586.8us
I20251026 14:59:21.694726 22553 catalog_manager.cc:5649] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:21.695816 22774 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:21.696485 22727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 853) took 1291 ms. Trace:
I20251026 14:59:21.537101 24677 consensus_queue.cc:1059] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.059s
I20251026 14:59:21.581236 24322 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:21.608767 24411 consensus_queue.cc:1059] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.102s
I20251026 14:59:21.705143 24595 raft_consensus.cc:2749] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251026 14:59:21.705349 24625 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:21.705813 24631 raft_consensus.cc:3055] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:21.706344 24631 raft_consensus.cc:740] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:21.707453 24631 consensus_queue.cc:260] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.708181 24630 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.709431 24629 raft_consensus.cc:2749] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:21.621691 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:21.658178 22803 raft_consensus.cc:1240] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:21.659068 22799 raft_consensus.cc:2889] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:21.581230 24476 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:21.711505 22806 raft_consensus.cc:1240] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:21.711746 22800 raft_consensus.cc:606] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:21.712181 22800 raft_consensus.cc:614] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251026 14:59:21.665799 22728 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
W20251026 14:59:21.533492 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 88713.5us
I20251026 14:59:21.605409 24638 raft_consensus.cc:2749] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:21.609704 24527 raft_consensus.cc:2804] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251026 14:59:21.674057 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:21.676733 22700 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:21.671821 22655 raft_consensus.cc:1240] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:21.696861 22727 rpcz_store.cc:276] 1026 14:59:20.405504 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:20.694376 (+288872us) service_pool.cc:225] Handling call
1026 14:59:20.694773 (+ 397us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:21.584974 (+890201us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:21.693122 (+108148us) spinlock_profiling.cc:243] Waited 890 ms on lock 0x7b5400076554. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:21.696429 (+ 3307us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2045888768}
I20251026 14:59:21.593803 24635 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:21.696962 22774 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:21.725696 22774 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:21.726848 22774 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:21.727375 22657 raft_consensus.cc:1275] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:21.693867 22807 raft_consensus.cc:2410] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251026 14:59:21.702697 22725 raft_consensus.cc:2393] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 5: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:21.729058 22807 raft_consensus.cc:1240] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
W20251026 14:59:21.703258 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:21.595302 24440 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:21.606710 22622 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:21.709019 24631 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:21.697626 24664 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:21.718106 22803 raft_consensus.cc:1275] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20251026 14:59:21.737144 22803 pending_rounds.cc:85] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821: Aborting all ops after (but not including) 2
I20251026 14:59:21.737450 22803 pending_rounds.cc:107] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251026 14:59:21.737700 22803 raft_consensus.cc:2889] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:21.602016 24405 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:21.720808 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 44221us
I20251026 14:59:21.597972 24678 raft_consensus.cc:2749] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:21.624368 24153 consensus_queue.cc:1048] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.108s
I20251026 14:59:21.751204 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:21.751825 22725 raft_consensus.cc:2468] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:21.717943 22771 leader_election.cc:304] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:21.693665 24634 raft_consensus.cc:740] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:21.757120 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 855) took 1317 ms. Trace:
I20251026 14:59:21.758561 22732 rpcz_store.cc:276] 1026 14:59:20.439664 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:20.834892 (+395228us) service_pool.cc:225] Handling call
1026 14:59:20.835284 (+ 392us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:21.501121 (+665837us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:21.756904 (+255783us) spinlock_profiling.cc:243] Waited 648 ms on lock 0x7b54000b95d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:21.757058 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1490578816}
I20251026 14:59:21.759631 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:21.728044 22774 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:21.598620 24384 consensus_queue.cc:1048] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.119s
I20251026 14:59:21.760706 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:21.761547 22732 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:21.761806 22774 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:21.743902 22803 raft_consensus.cc:1275] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:21.747480 22773 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:21.749089 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:21.750105 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:21.622761 24579 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.753890 22807 raft_consensus.cc:1240] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:21.754251 24479 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:21.754315 24481 raft_consensus.cc:3055] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:21.754725 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:21.755759 24663 log.cc:927] Time spent T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.578s user 0.006s sys 0.000s
I20251026 14:59:21.756369 22729 raft_consensus.cc:1275] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20251026 14:59:21.756872 22802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1033) took 2065 ms. Trace:
I20251026 14:59:21.736821 24677 raft_consensus.cc:2749] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:21.757504 22808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1010) took 1084 ms. Trace:
I20251026 14:59:21.741070 24629 raft_consensus.cc:2749] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:21.751456 24638 raft_consensus.cc:2749] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:59:21.753336 24103 raft_consensus.cc:2749] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:21.733851 24476 raft_consensus.cc:2804] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:21.654026 24671 consensus_queue.cc:1059] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.100s
I20251026 14:59:21.734967 24440 raft_consensus.cc:3060] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:21.734313 24628 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:21.783748 22808 rpcz_store.cc:276] 1026 14:59:20.672701 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:21.484529 (+811828us) service_pool.cc:225] Handling call
1026 14:59:21.731567 (+247038us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:21.731740 (+ 173us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:21.731764 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:21.731804 (+ 40us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:21.731831 (+ 27us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:21.731878 (+ 47us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:21.757347 (+ 25469us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b54000c6550. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:21.757443 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":567320448}
I20251026 14:59:21.719264 24527 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:21.781693 24322 raft_consensus.cc:3060] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:59:21.796089 24626 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.786278 24479 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:59:21.799713 24518 raft_consensus.cc:3055] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:21.783350 24625 raft_consensus.cc:2804] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:21.803210 24667 raft_consensus.cc:3055] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Stepping down as leader of term 1
W20251026 14:59:21.804756 24670 log.cc:927] Time spent T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.271s user 0.000s sys 0.005s
I20251026 14:59:21.800611 24403 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.726197 24578 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.780448 24481 raft_consensus.cc:740] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:21.778561 24678 raft_consensus.cc:2749] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:21.815716 24527 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:21.782810 24455 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:21.752537 24664 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.808310 24336 raft_consensus.cc:3055] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
W20251026 14:59:21.811853 24681 log.cc:927] Time spent T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.282s user 0.002s sys 0.000s
I20251026 14:59:21.767592 24405 raft_consensus.cc:2749] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:21.783481 24595 consensus_queue.cc:1048] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.037s
I20251026 14:59:21.856097 24595 consensus_queue.cc:1243] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer df3a0d1619ee48e8ab68ee0543a72a29 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251026 14:59:21.821663 24634 consensus_queue.cc:260] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.821601 24554 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.823323 24625 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:21.822687 24476 raft_consensus.cc:697] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:21.832995 24579 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:21.857218 24634 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:21.821195 24518 raft_consensus.cc:740] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:21.858249 24476 consensus_queue.cc:237] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.858469 24518 consensus_queue.cc:260] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.859387 24554 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:21.859766 24518 raft_consensus.cc:3060] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:21.861831 24664 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:21.827315 24532 consensus_queue.cc:1059] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20251026 14:59:21.841531 24481 consensus_queue.cc:260] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:21.865172 24578 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:21.866848 24662 log.cc:927] Time spent T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.689s user 0.000s sys 0.004s
I20251026 14:59:21.857242 24625 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:21.855752 24630 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:21.844836 24626 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:21.843298 24479 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:21.823716 24667 raft_consensus.cc:740] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:21.875928 24527 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:21.741137 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 134514us
I20251026 14:59:21.972610 24625 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.973919 22622 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:21.755712 22652 raft_consensus.cc:1275] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251026 14:59:21.908063 22604 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet fbb846c2db3b4f44b2245a7493b36c7c, TS df3a0d1619ee48e8ab68ee0543a72a29 -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.973390 24635 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.914943 22726 raft_consensus.cc:2368] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6.
W20251026 14:59:21.914752 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 154181us
I20251026 14:59:21.978989 24593 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:21.916127 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:21.980147 24593 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.982646 22654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1022) took 1127 ms. Trace:
I20251026 14:59:21.982769 22725 raft_consensus.cc:1275] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:21.921789 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:21.983220 22654 rpcz_store.cc:276] 1026 14:59:20.854796 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:21.543157 (+688361us) service_pool.cc:225] Handling call
1026 14:59:21.543420 (+ 263us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:21.693844 (+150424us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:21.982000 (+288156us) spinlock_profiling.cc:243] Waited 45.6 ms on lock 0x7b540009e2d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:21.982403 (+ 403us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":104989440}
I20251026 14:59:21.923563 22728 raft_consensus.cc:1240] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20251026 14:59:21.770027 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:21.989135 22730 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:21.991925 22732 raft_consensus.cc:2468] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:21.925241 22731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1249) took 1042 ms. Trace:
I20251026 14:59:21.981169 24403 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:21.779188 22774 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:21.999307 24634 raft_consensus.cc:2749] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20251026 14:59:21.999818 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 220606us
W20251026 14:59:21.945168 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:22.003086 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:21.757120 22799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1004) took 1110 ms. Trace:
I20251026 14:59:22.004241 22799 rpcz_store.cc:276] 1026 14:59:20.646964 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:20.647189 (+ 225us) service_pool.cc:225] Handling call
1026 14:59:20.671032 (+ 23843us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:21.711451 (+1040419us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:21.756936 (+ 45485us) spinlock_profiling.cc:243] Waited 783 ms on lock 0x7b54001b80d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:21.757058 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1800040704}
I20251026 14:59:22.004994 24322 raft_consensus.cc:2749] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 14:59:21.780675 22802 rpcz_store.cc:276] 1026 14:59:19.691213 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:20.342323 (+651110us) service_pool.cc:225] Handling call
1026 14:59:21.323828 (+981505us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:21.424089 (+100261us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:21.756648 (+332559us) spinlock_profiling.cc:243] Waited 947 ms on lock 0x7b540038aed0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:21.756809 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2178081024}
I20251026 14:59:21.764111 22803 raft_consensus.cc:1275] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 3. (term mismatch)
I20251026 14:59:21.945798 22804 raft_consensus.cc:2393] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 6: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
W20251026 14:59:21.946225 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 210238us
I20251026 14:59:22.008494 22803 pending_rounds.cc:85] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821: Aborting all ops after (but not including) 2
I20251026 14:59:22.009038 22803 pending_rounds.cc:107] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20251026 14:59:22.009407 22803 raft_consensus.cc:2889] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:21.946791 22808 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733"
dest_uuid: "0d684445287b450a8f738b6ad4b59821"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:44012
I20251026 14:59:22.010730 22808 raft_consensus.cc:606] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:22.011231 22808 raft_consensus.cc:614] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20251026 14:59:22.011389 23370 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:21.948082 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 214691us
I20251026 14:59:21.993470 22731 rpcz_store.cc:276] 1026 14:59:20.882919 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:21.363801 (+480882us) service_pool.cc:225] Handling call
1026 14:59:21.384092 (+ 20291us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:21.384419 (+ 327us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:21.384438 (+ 19us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:21.398933 (+ 14495us) log.cc:844] Serialized 52 byte log entry
1026 14:59:21.529176 (+130243us) raft_consensus.cc:1645] Marking committed up to 1
1026 14:59:21.529219 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:21.529321 (+ 102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:21.924214 (+394893us) raft_consensus.cc:1685] finished
1026 14:59:21.924260 (+ 46us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:21.924980 (+ 720us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3000,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":11000,"thread_start_us":87831,"threads_started":1,"wal-append.queue_time_us":106663}
I20251026 14:59:21.791626 24332 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:21.949537 24336 raft_consensus.cc:740] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:21.823964 24353 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:21.960920 24032 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:22.014617 24336 consensus_queue.cc:260] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.016158 24032 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.017794 24689 consensus_queue.cc:1059] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251026 14:59:21.877251 24667 consensus_queue.cc:260] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.019436 24032 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:22.021595 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 1246) took 1139 ms. Trace:
I20251026 14:59:22.022044 22726 rpcz_store.cc:276] 1026 14:59:20.881996 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:21.032588 (+150592us) service_pool.cc:225] Handling call
1026 14:59:22.021302 (+988714us) spinlock_profiling.cc:243] Waited 879 ms on lock 0x7b54000b7cd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:22.021510 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2022610816}
I20251026 14:59:21.961822 24115 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.975868 24536 raft_consensus.cc:2804] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:21.976047 24625 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:21.976547 22604 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet fbb846c2db3b4f44b2245a7493b36c7c, TS df3a0d1619ee48e8ab68ee0543a72a29 -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:21.914206 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:21.964461 24103 consensus_queue.cc:1059] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20251026 14:59:21.806524 22806 raft_consensus.cc:1240] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 1. Current term is 2. Ops: []
I20251026 14:59:22.018159 24455 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:22.040143 22808 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570"
dest_uuid: "0d684445287b450a8f738b6ad4b59821"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:44012
I20251026 14:59:22.040443 24455 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.040934 24476 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:22.041776 24476 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.043715 24476 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:22.045881 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 44942.7us
I20251026 14:59:22.046515 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:22.047482 22657 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:22.049669 24103 raft_consensus.cc:3055] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:22.049794 24686 raft_consensus.cc:3055] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:22.050283 24103 raft_consensus.cc:740] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:22.050415 24686 raft_consensus.cc:740] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:22.050803 24455 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:22.052168 24481 raft_consensus.cc:3060] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:22.051854 24103 consensus_queue.cc:260] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:21.971441 24527 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:22.016973 24336 raft_consensus.cc:3060] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:22.034832 23370 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:22.037468 22733 raft_consensus.cc:1240] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:22.052013 24686 consensus_queue.cc:260] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.040984 22808 raft_consensus.cc:606] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:22.055382 24593 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:22.057930 24536 raft_consensus.cc:697] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:22.060735 24536 consensus_queue.cc:237] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.061990 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:22.064906 22808 raft_consensus.cc:614] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251026 14:59:22.064942 24476 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:22.068176 24667 raft_consensus.cc:3060] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 1 FOLLOWER]: Advancing to term 2
I20251026 14:59:22.068328 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:22.068918 24103 raft_consensus.cc:3060] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:22.070791 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:22.073633 22734 raft_consensus.cc:2393] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 5: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
W20251026 14:59:22.074450 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 78104.1us
I20251026 14:59:22.075524 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:22.075989 22622 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:22.074498 22650 raft_consensus.cc:1275] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:22.078876 22654 raft_consensus.cc:2393] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:22.079419 24532 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:22.080372 24532 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:22.083197 24532 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:22.083568 24686 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:22.067924 24476 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.074517 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:22.086875 24476 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:22.088677 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:22.089810 22731 raft_consensus.cc:2393] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 6: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:22.089856 22622 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:22.091032 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20251026 14:59:22.095044 24527 raft_consensus.cc:2804] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:22.095605 24527 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:22.096117 24527 raft_consensus.cc:3060] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:22.098208 24635 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:22.100116 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:22.108032 22622 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:22.111732 24031 raft_consensus.cc:2749] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:22.111743 24153 consensus_queue.cc:1059] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20251026 14:59:22.114266 24115 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:22.121191 24527 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.122489 24032 consensus_queue.cc:1048] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:22.138862 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:22.123466 24532 consensus_queue.cc:1059] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251026 14:59:22.123605 24031 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:22.140779 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:59:22.140591 24031 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.141559 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:22.142057 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:59:22.142572 22726 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:22.143380 24031 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:22.144924 24031 raft_consensus.cc:3055] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:22.058542 22774 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:22.125660 24403 consensus_queue.cc:1059] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251026 14:59:22.125648 24418 consensus_queue.cc:1048] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
W20251026 14:59:22.146152 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 97023.2us
I20251026 14:59:22.125721 24353 raft_consensus.cc:3055] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:22.147332 24353 raft_consensus.cc:740] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:22.148873 22774 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:22.130790 24332 consensus_queue.cc:1048] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:59:22.145332 24031 raft_consensus.cc:740] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:22.131115 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:22.131409 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:22.151394 24031 consensus_queue.cc:260] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.134133 24385 consensus_queue.cc:1048] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:22.152383 22731 raft_consensus.cc:2393] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 5: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:22.152325 22725 raft_consensus.cc:2468] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:22.137611 22624 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:22.138051 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:22.154027 22808 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:22.154201 24385 raft_consensus.cc:2804] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:22.154781 24385 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:22.156103 24385 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:22.148602 24353 consensus_queue.cc:260] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.157434 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:22.158223 24353 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:22.159036 24031 raft_consensus.cc:3060] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:22.158253 22731 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 7
I20251026 14:59:22.149556 24625 raft_consensus.cc:2749] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:22.146173 24403 raft_consensus.cc:3055] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:22.165570 24403 raft_consensus.cc:740] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:22.166671 24403 consensus_queue.cc:260] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.168015 24403 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:22.128077 24153 consensus_queue.cc:1059] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:59:22.170539 24153 raft_consensus.cc:3055] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:22.171038 24153 raft_consensus.cc:740] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:22.170884 24332 consensus_queue.cc:1059] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:59:22.151602 24593 raft_consensus.cc:2749] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:22.172303 24332 raft_consensus.cc:3055] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:22.172689 24153 consensus_queue.cc:260] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.173586 24032 consensus_queue.cc:1048] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:59:22.174322 24153 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:22.139685 22652 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:22.132344 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:22.175609 22657 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:22.176259 24589 raft_consensus.cc:2764] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20251026 14:59:22.172724 24332 raft_consensus.cc:740] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:22.156093 22730 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:22.178714 24332 consensus_queue.cc:260] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.180756 24332 raft_consensus.cc:3060] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:22.183822 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:22.184952 22733 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:22.187330 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:22.193338 24635 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:22.194265 24635 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:22.193485 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:22.196763 24635 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:22.151867 22774 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:22.164785 24385 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.210614 22808 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:22.212286 22726 raft_consensus.cc:2468] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
W20251026 14:59:22.206298 24544 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.649s user 0.013s sys 0.010s
W20251026 14:59:22.216177 24544 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.667s user 0.013s sys 0.010s
I20251026 14:59:22.217346 22801 raft_consensus.cc:1240] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:22.217583 24610 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:22.218235 24610 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.217353 24440 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.220450 24610 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:22.233317 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:22.238967 22726 raft_consensus.cc:2393] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 7: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:22.240891 22808 raft_consensus.cc:1240] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:22.223062 24322 raft_consensus.cc:2749] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:22.253475 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:22.258097 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 106395us
I20251026 14:59:22.260583 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:22.281648 22801 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
W20251026 14:59:22.283915 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:22.284117 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:22.284776 22774 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:22.286315 24322 raft_consensus.cc:2749] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:22.287377 24440 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:22.287880 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:22.288411 24385 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:22.289975 24610 consensus_queue.cc:1048] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:59:22.287424 22806 raft_consensus.cc:1240] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:22.284971 22801 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:22.291294 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:22.297993 22807 raft_consensus.cc:2393] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 6: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:22.298653 22731 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:22.300071 24354 consensus_queue.cc:1048] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20251026 14:59:22.302615 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:22.303565 23399 raft_consensus.cc:2468] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:22.304296 24527 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:22.304890 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:22.314584 22808 raft_consensus.cc:2468] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:22.293529 24322 consensus_queue.cc:1059] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20251026 14:59:22.311873 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:22.312361 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:22.317531 22622 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:22.318662 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:22.319828 22807 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:22.320880 24629 consensus_queue.cc:1059] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:59:22.321842 24687 raft_consensus.cc:3055] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:22.322178 24544 raft_consensus.cc:3055] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:22.322314 24687 raft_consensus.cc:740] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:22.322667 24544 raft_consensus.cc:740] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:22.323760 24322 raft_consensus.cc:3055] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:22.324227 24322 raft_consensus.cc:740] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:22.323884 24544 consensus_queue.cc:260] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.325342 24322 consensus_queue.cc:260] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:22.329178 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:22.329356 22804 raft_consensus.cc:2393] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:22.329891 22806 raft_consensus.cc:2468] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
W20251026 14:59:22.332186 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:22.333407 24527 raft_consensus.cc:2804] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:22.334128 24527 raft_consensus.cc:697] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:22.335506 24527 consensus_queue.cc:237] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.337311 22622 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:22.337318 24385 consensus_queue.cc:1059] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20251026 14:59:22.337891 24332 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:22.338203 24440 raft_consensus.cc:2749] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:22.338519 24385 raft_consensus.cc:3055] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:22.338968 24385 raft_consensus.cc:740] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:22.338732 24332 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.339840 22622 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:22.340732 24440 raft_consensus.cc:2804] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:22.339926 24385 consensus_queue.cc:260] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.341110 24440 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:22.341243 24332 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:22.341670 24440 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:22.341758 24385 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:22.358816 22774 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:22.361742 22726 raft_consensus.cc:1240] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20251026 14:59:22.369709 24625 raft_consensus.cc:2804] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:22.369931 22774 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:22.370308 24625 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:22.370764 24625 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:22.371013 24322 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:22.373564 24593 raft_consensus.cc:2764] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20251026 14:59:22.374001 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:22.374833 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:22.375280 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:22.376082 22799 raft_consensus.cc:2393] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 6: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:22.375429 22806 raft_consensus.cc:3060] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
W20251026 14:59:22.377454 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 45203.3us
I20251026 14:59:22.359922 24354 consensus_queue.cc:1048] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20251026 14:59:22.381625 24379 consensus_queue.cc:1059] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251026 14:59:22.371488 22774 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:22.329298 24544 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:22.384821 24625 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.374873 22803 raft_consensus.cc:2393] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 3: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:22.385977 22622 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:22.387032 24332 raft_consensus.cc:2804] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:22.387105 22622 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:22.387480 24332 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:22.387892 24332 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:22.393571 22806 raft_consensus.cc:2468] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
W20251026 14:59:22.394308 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:22.403365 24536 raft_consensus.cc:2804] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:22.403852 24536 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:22.397464 24593 raft_consensus.cc:2804] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:22.404202 24536 raft_consensus.cc:3060] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:22.404428 24593 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:22.404937 24593 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:22.411933 22622 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:22.419056 24687 consensus_queue.cc:260] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:22.420944 24687 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:22.395679 24625 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:22.398016 24354 consensus_queue.cc:1048] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20251026 14:59:22.422861 22622 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:22.437528 24336 raft_consensus.cc:2804] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:22.437985 24336 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:22.439023 24336 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:22.439244 22622 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:22.438277 22700 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:22.422641 24031 raft_consensus.cc:2804] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:22.414811 24332 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:22.451957 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:22.458339 22799 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:22.463330 24625 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:22.464010 24625 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:59:22.466852 24625 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:22.467507 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 44901.2us
I20251026 14:59:22.468072 24587 raft_consensus.cc:2804] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:22.468425 22700 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:22.468695 24587 raft_consensus.cc:697] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:22.469383 24610 raft_consensus.cc:2804] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:22.469987 24610 raft_consensus.cc:697] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:22.448205 24686 raft_consensus.cc:2804] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:22.446494 24031 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:22.504297 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:22.508219 22657 raft_consensus.cc:1240] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:22.490356 24031 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:22.455327 24536 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.511916 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:22.517673 22654 raft_consensus.cc:2393] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 5: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:22.529125 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:22.524597 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:22.511898 24686 raft_consensus.cc:697] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:22.534425 22659 raft_consensus.cc:2368] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 5.
I20251026 14:59:22.534262 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:22.538568 22657 raft_consensus.cc:2393] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 5: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
W20251026 14:59:22.540585 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 78938.4us
I20251026 14:59:22.549432 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:22.554770 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:22.555063 24536 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:22.555792 22650 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:22.556169 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:22.527070 24153 consensus_queue.cc:1048] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.023s
I20251026 14:59:22.551713 24587 consensus_queue.cc:237] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.559158 24332 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:22.519590 24440 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.561379 24440 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:22.562045 24702 log.cc:927] Time spent T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20251026 14:59:22.564344 24336 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:22.565466 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:22.566615 22652 raft_consensus.cc:2368] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6.
I20251026 14:59:22.566787 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:22.570885 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:22.494197 24610 consensus_queue.cc:237] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.576170 22654 raft_consensus.cc:1275] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:22.580047 24336 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:22.582419 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:22.583381 22650 raft_consensus.cc:2393] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:22.584134 24354 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:22.585067 22658 raft_consensus.cc:1240] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:22.584848 24354 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.587080 24354 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:22.597599 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:22.599905 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:22.599854 24686 consensus_queue.cc:237] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.601666 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:22.602367 22732 raft_consensus.cc:2393] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 6: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:22.602196 24354 consensus_queue.cc:1059] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:59:22.603153 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:22.603461 24635 raft_consensus.cc:3055] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:22.603622 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:22.603914 24635 raft_consensus.cc:740] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:22.604470 22731 raft_consensus.cc:2393] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 5: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:22.604692 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:22.604790 24635 consensus_queue.cc:260] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.606325 24635 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:22.604503 22658 raft_consensus.cc:1275] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251026 14:59:22.606240 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:22.601048 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:22.603766 22728 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:22.612264 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:22.613267 22727 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:22.617754 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:22.621628 22697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:22.630468 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:22.631240 22650 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:22.631264 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:22.635623 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:22.637065 22659 raft_consensus.cc:1240] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251026 14:59:22.644905 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:22.646986 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:22.623157 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:22.649122 22803 raft_consensus.cc:2393] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:22.656663 22700 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:22.589059 24593 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:22.660246 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:22.667186 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:22.661733 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:22.664484 22658 raft_consensus.cc:2376] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5. Re-sending same reply.
I20251026 14:59:22.686993 22650 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:22.677101 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:22.661439 24354 raft_consensus.cc:2804] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:22.689846 22654 raft_consensus.cc:1240] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20251026 14:59:22.689836 22806 raft_consensus.cc:1240] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6. Ops: []
W20251026 14:59:22.691305 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 89103.5us
I20251026 14:59:22.684798 24336 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:22.695399 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:22.691982 22624 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:22.700511 22799 raft_consensus.cc:1240] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:22.699461 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:22.703409 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:22.705188 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:22.705964 22803 raft_consensus.cc:2376] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6. Re-sending same reply.
I20251026 14:59:22.716463 22804 raft_consensus.cc:2393] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:22.717060 22654 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:22.694022 24354 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:22.729534 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:22.732295 22658 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:22.736054 22657 raft_consensus.cc:1240] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20251026 14:59:22.724294 24354 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:22.738816 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:22.729475 24418 raft_consensus.cc:2749] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:22.750816 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:22.758813 22653 raft_consensus.cc:2468] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:22.733703 24336 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.762032 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:22.762315 24336 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:22.762887 22650 raft_consensus.cc:2393] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 6: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:22.763202 22622 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:22.763696 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:22.765578 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:22.766146 22804 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:22.766348 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:22.767168 22799 raft_consensus.cc:2393] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 6: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:22.767823 22656 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:22.768929 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:22.771986 22806 raft_consensus.cc:3060] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:22.773965 24593 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:59:22.774312 24332 raft_consensus.cc:2804] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:22.774716 24332 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:22.775132 24332 raft_consensus.cc:3060] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:22.775168 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:22.778992 22733 raft_consensus.cc:2468] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
W20251026 14:59:22.782604 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 123884us
W20251026 14:59:22.781383 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 43534.9us
I20251026 14:59:22.790717 22806 raft_consensus.cc:2468] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:22.791806 22773 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:22.794600 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:22.793488 22624 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:22.795296 22652 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:22.797135 22653 raft_consensus.cc:1240] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20251026 14:59:22.799576 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:22.799998 22622 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:22.800210 22806 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:22.798259 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:22.803155 22803 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:22.801813 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:22.806483 22799 raft_consensus.cc:1275] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251026 14:59:22.808427 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:22.804930 24418 raft_consensus.cc:2804] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:22.821928 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:22.822829 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:22.823709 22800 raft_consensus.cc:2393] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:22.826861 22804 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:22.830471 22622 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:22.817888 24418 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:22.831540 24418 raft_consensus.cc:3060] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:22.831290 24723 consensus_queue.cc:1048] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.029s
I20251026 14:59:22.788769 24031 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.840276 22654 raft_consensus.cc:2468] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
W20251026 14:59:22.842213 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 48955us
I20251026 14:59:22.824394 24593 raft_consensus.cc:2804] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:22.801429 24587 consensus_queue.cc:1059] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20251026 14:59:22.848662 22799 raft_consensus.cc:1275] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:22.849985 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:22.853117 22800 raft_consensus.cc:2393] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 7: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:22.854655 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:22.822011 24610 consensus_queue.cc:1048] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.026s
I20251026 14:59:22.858494 22799 raft_consensus.cc:1275] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:22.860733 22655 raft_consensus.cc:2368] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6.
I20251026 14:59:22.858690 22807 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
W20251026 14:59:22.862890 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 79501.7us
I20251026 14:59:22.863497 22700 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:22.871701 22654 raft_consensus.cc:1275] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:22.877583 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:22.882443 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:22.865640 24593 raft_consensus.cc:697] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:22.861481 24527 raft_consensus.cc:2804] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:22.873855 24532 consensus_queue.cc:1048] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20251026 14:59:22.883481 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:22.894418 24031 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:22.896818 22652 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:22.897400 22726 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:22.896420 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:22.875504 24635 raft_consensus.cc:2804] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:22.891634 22697 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:22.899739 22650 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:22.901151 22805 raft_consensus.cc:2393] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:22.907866 22773 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:22.908592 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:22.909189 22807 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:22.884894 24587 consensus_queue.cc:1059] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20251026 14:59:22.900867 24635 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:22.917263 22805 raft_consensus.cc:1275] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251026 14:59:22.919148 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:22.913579 24115 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20251026 14:59:22.924043 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:22.901593 24726 consensus_queue.cc:1059] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20251026 14:59:22.925266 22800 raft_consensus.cc:2393] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:22.925565 22803 raft_consensus.cc:2468] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:22.917764 24635 raft_consensus.cc:3060] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:22.929886 24593 consensus_queue.cc:237] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.932567 24332 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:22.933133 22658 raft_consensus.cc:2468] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:22.940734 24418 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.929929 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:22.910645 24727 consensus_queue.cc:1059] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20251026 14:59:22.930130 24730 consensus_queue.cc:1048] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20251026 14:59:22.956782 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 69378.7us
I20251026 14:59:22.925130 24115 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:22.961122 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:22.958016 22773 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:22.957948 22774 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:22.963840 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:22.980443 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:22.982713 22659 raft_consensus.cc:2468] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:22.950660 24153 raft_consensus.cc:2749] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:22.927249 24354 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:22.977798 24115 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
W20251026 14:59:22.993503 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 93433.3us
I20251026 14:59:22.997018 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:22.997946 22659 raft_consensus.cc:2368] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6.
I20251026 14:59:22.997972 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:22.999315 22650 raft_consensus.cc:2393] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 6: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:23.000061 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:23.002182 22807 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:23.005893 22658 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:23.007023 22622 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:23.009083 22624 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:23.020684 22622 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:23.022243 24735 raft_consensus.cc:2804] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:23.022981 24735 raft_consensus.cc:697] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:23.024291 24735 consensus_queue.cc:237] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:23.026629 22624 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:23.027428 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 69697.1us
I20251026 14:59:23.028095 22774 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:23.025767 24726 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:23.013502 24593 raft_consensus.cc:2804] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:23.025972 24738 raft_consensus.cc:3055] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:23.026609 24532 raft_consensus.cc:2749] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:23.028836 24536 raft_consensus.cc:2804] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:23.031280 24322 raft_consensus.cc:2749] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:23.032400 24587 raft_consensus.cc:3055] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:23.033243 24385 raft_consensus.cc:2749] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:23.029063 24727 consensus_queue.cc:1243] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer df3a0d1619ee48e8ab68ee0543a72a29 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251026 14:59:23.067292 22624 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:23.072420 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:23.073367 24544 raft_consensus.cc:2749] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:23.037678 24738 raft_consensus.cc:740] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:23.060052 24536 raft_consensus.cc:697] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:23.060681 24587 raft_consensus.cc:740] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:23.063668 24593 raft_consensus.cc:697] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:23.101325 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:23.102205 22658 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:23.104563 24732 raft_consensus.cc:2749] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:23.090808 24332 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:23.104379 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:23.108661 24587 consensus_queue.cc:260] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:23.110615 24536 consensus_queue.cc:237] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:23.111316 24587 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
W20251026 14:59:23.113696 24721 log.cc:927] Time spent T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.142s user 0.000s sys 0.001s
I20251026 14:59:23.118590 24593 consensus_queue.cc:237] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:23.074662 24738 consensus_queue.cc:260] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:23.136229 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 67407us
W20251026 14:59:23.133100 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 175178us
I20251026 14:59:23.136708 24532 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:23.138976 24532 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:23.127090 24354 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:23.142422 22624 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
W20251026 14:59:23.145808 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 278495us
W20251026 14:59:23.142577 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 114530us
I20251026 14:59:23.148972 24532 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:23.152163 22773 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:23.135412 22658 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:23.154469 22700 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:23.155745 22774 leader_election.cc:400] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 5.
I20251026 14:59:23.155619 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:23.155352 24336 raft_consensus.cc:2804] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:23.158829 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:23.143834 24527 raft_consensus.cc:697] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:23.159611 22804 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:23.163080 22622 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:23.163851 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:23.164821 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:23.166596 24336 raft_consensus.cc:697] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:23.167671 24635 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:23.167590 24336 consensus_queue.cc:237] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:23.170545 22697 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:23.156879 24354 raft_consensus.cc:2804] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election won for term 7
W20251026 14:59:23.178054 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:23.160946 24738 raft_consensus.cc:3060] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:23.182222 22773 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:23.186914 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:23.187516 22727 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:23.193126 24385 raft_consensus.cc:2804] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:23.193686 24385 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:23.194077 24385 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:23.196201 24726 raft_consensus.cc:2749] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:23.197933 24736 raft_consensus.cc:2804] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:23.198486 24736 raft_consensus.cc:697] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:23.199353 24736 consensus_queue.cc:237] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:23.200538 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 45918.5us
W20251026 14:59:23.230993 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 79543.4us
I20251026 14:59:23.231848 22774 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:23.201182 24723 raft_consensus.cc:3055] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:23.239792 22697 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:23.241405 24742 raft_consensus.cc:2804] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:23.218489 24732 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:23.242811 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:23.247243 22730 raft_consensus.cc:1275] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251026 14:59:23.241104 22697 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:23.250733 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:23.255388 22734 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:23.230614 24527 consensus_queue.cc:237] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:23.257845 22804 raft_consensus.cc:1275] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:23.231451 24354 raft_consensus.cc:697] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:23.257449 24752 raft_consensus.cc:2749] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:23.265913 22730 raft_consensus.cc:3055] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:23.269052 22730 raft_consensus.cc:740] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:23.241865 24742 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:23.244668 24723 raft_consensus.cc:740] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:23.247263 24726 raft_consensus.cc:2749] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:23.279707 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:23.560761 22731 raft_consensus.cc:2393] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 5: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:23.250907 24732 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:23.291131 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 44251us
W20251026 14:59:23.344368 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 112624us
I20251026 14:59:23.566473 22733 raft_consensus.cc:2410] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20251026 14:59:23.422972 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:23.281592 24742 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:23.282644 24740 raft_consensus.cc:2749] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:23.268669 24332 consensus_queue.cc:1048] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.097s
I20251026 14:59:23.278199 24727 raft_consensus.cc:2749] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251026 14:59:23.488376 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 205517us
I20251026 14:59:23.481124 24686 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:23.574734 24753 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:23.277827 22730 consensus_queue.cc:260] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:23.516053 22734 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:23.554857 24587 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:23.559504 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:23.284189 24354 consensus_queue.cc:237] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:23.478597 24403 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:23.569681 22727 raft_consensus.cc:2468] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:23.572427 22774 leader_election.cc:400] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6.
I20251026 14:59:23.593678 22774 leader_election.cc:403] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
W20251026 14:59:23.535871 24719 log.cc:927] Time spent T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.695s user 0.008s sys 0.017s
W20251026 14:59:23.574576 24757 log.cc:927] Time spent T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.073s user 0.005s sys 0.000s
I20251026 14:59:23.598510 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:59:23.578235 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:23.589552 22725 raft_consensus.cc:2393] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 6: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:23.560606 24736 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:23.603562 22659 raft_consensus.cc:1275] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:23.622880 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:23.596172 24726 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:23.603039 24741 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:23.643412 22658 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:23.609977 24727 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:23.596700 24686 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:23.615195 24593 raft_consensus.cc:2749] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6.
I20251026 14:59:23.605304 24587 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:23.602640 24403 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:23.627076 24536 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:23.664875 24686 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:23.602386 24753 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:23.667629 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 75236.1us
W20251026 14:59:23.674160 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:23.661063 24745 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:23.637986 24726 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:23.623559 24736 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:23.659878 24727 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:23.730592 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 132611us
I20251026 14:59:23.690219 24593 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:23.735200 22697 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:23.738520 22700 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:23.679260 24536 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:23.727597 23399 consensus_meta.cc:287] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1: Time spent flushing consensus metadata: real 0.621s user 0.011s sys 0.011s
W20251026 14:59:23.729377 24115 consensus_meta.cc:287] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.716s user 0.020s sys 0.001s
I20251026 14:59:23.661500 24741 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:23.732312 24385 consensus_meta.cc:287] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.537s user 0.009s sys 0.007s
I20251026 14:59:23.736974 22624 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:23.745642 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:23.746582 22731 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:23.747434 22624 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:23.754457 22653 raft_consensus.cc:1240] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5. Ops: []
W20251026 14:59:23.731158 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 252717us
W20251026 14:59:23.765565 24762 log.cc:927] Time spent T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.195s user 0.002s sys 0.000s
I20251026 14:59:23.766094 22732 raft_consensus.cc:1275] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20251026 14:59:23.744232 24738 consensus_meta.cc:287] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.553s user 0.013s sys 0.012s
W20251026 14:59:23.775003 24764 log.cc:927] Time spent T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20251026 14:59:23.775668 24767 consensus_queue.cc:1048] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:59:23.776618 24440 raft_consensus.cc:2749] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251026 14:59:23.778504 24760 log.cc:927] Time spent T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.143s user 0.000s sys 0.001s
W20251026 14:59:23.779887 24761 log.cc:927] Time spent T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20251026 14:59:23.783202 24727 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:23.784004 24727 raft_consensus.cc:2749] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:23.786885 24753 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
W20251026 14:59:23.791635 24755 log.cc:927] Time spent T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.248s user 0.000s sys 0.001s
I20251026 14:59:23.775033 22732 raft_consensus.cc:1275] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251026 14:59:23.804430 22726 raft_consensus.cc:2410] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251026 14:59:23.743124 23399 raft_consensus.cc:2468] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:23.757109 24740 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:23.745122 24766 raft_consensus.cc:2804] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:23.718226 24745 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:23.843207 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:23.845764 22659 raft_consensus.cc:2410] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20251026 14:59:23.842883 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:23.848187 22653 raft_consensus.cc:2393] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:23.747725 24610 consensus_queue.cc:1048] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.086s
I20251026 14:59:23.852562 24726 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:23.769873 24732 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:23.854488 24536 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:23.849174 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:23.855934 24736 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:23.856551 24736 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:23.857070 22802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1046) took 1437 ms. Trace:
I20251026 14:59:23.857038 24736 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:23.857589 24766 raft_consensus.cc:697] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:23.784782 24727 consensus_queue.cc:1059] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251026 14:59:23.752574 24593 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:23.859316 24736 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:23.750070 24115 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:23.789191 24418 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:23.751010 24385 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:23.872051 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:23.873188 22655 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:23.838018 24772 raft_consensus.cc:2749] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:23.881114 22700 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:23.895470 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 168304us
I20251026 14:59:23.899605 22800 raft_consensus.cc:1240] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:23.870126 24736 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:23.870313 24587 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:23.858968 24766 consensus_queue.cc:237] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:23.993184 22622 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:23.919584 24726 raft_consensus.cc:2804] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:24.004441 22803 raft_consensus.cc:1275] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:24.005833 24593 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:24.007001 24593 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:24.007326 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:24.007494 24593 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.008054 22806 raft_consensus.cc:1275] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:59:24.008104 22803 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:24.008522 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:24.009101 22808 raft_consensus.cc:3060] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:24.009181 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:23.857606 22802 rpcz_store.cc:276] 1026 14:59:22.311262 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:22.313327 (+ 2065us) service_pool.cc:225] Handling call
1026 14:59:22.323146 (+ 9819us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:22.327157 (+ 4011us) raft_consensus.cc:1534] Early marking committed up to index 3
1026 14:59:22.457431 (+130274us) log.cc:844] Serialized 14 byte log entry
1026 14:59:22.677193 (+219762us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:22.742604 (+ 65411us) log.cc:844] Serialized 26 byte log entry
1026 14:59:22.762910 (+ 20306us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:22.762956 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:22.763035 (+ 79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:23.430118 (+667083us) spinlock_profiling.cc:243] Waited 9.28 ms on lock 0x7b440002304c. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8cdfc0905 00007fd8cdfc1322 00007fd8d0d0a231 00007fd8d0d05b5a 00007fd8d0cf991e 00007fd8d0cf8c66 00007fd8d0cf8888 00007fd8d0d022c1 00007fd8d0d02279 00007fd8d0d02209
1026 14:59:23.730545 (+300427us) raft_consensus.cc:1685] finished
1026 14:59:23.736141 (+ 5596us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:23.748738 (+ 12597us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2071,"prepare.queue_time_us":19943,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":4345,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":5737,"spinlock_wait_cycles":21344000,"thread_start_us":253998,"threads_started":2,"wal-append.queue_time_us":295753}
I20251026 14:59:24.011211 22806 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:59:24.012416 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:24.013311 22803 raft_consensus.cc:2468] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:59:23.850309 24740 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:23.984779 24727 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:24.016149 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 267678us
I20251026 14:59:24.021898 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:24.032280 22804 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:23.993434 24336 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:24.033450 22622 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:23.995317 24403 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:24.002269 24726 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:24.022800 24772 raft_consensus.cc:2804] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:24.015101 24587 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:59:23.997042 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 194906us
W20251026 14:59:24.053614 24765 log.cc:927] Time spent T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.281s user 0.001s sys 0.000s
I20251026 14:59:24.075429 24589 raft_consensus.cc:2804] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:24.076288 24589 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:24.076829 24589 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:24.079305 22808 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:24.081715 22697 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:23.999480 24736 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:24.089442 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 56703.7us
I20251026 14:59:24.092247 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:24.092316 22622 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:24.052361 24726 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:24.049942 24741 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:24.102742 22731 raft_consensus.cc:2393] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 3: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:24.105244 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:24.049757 24686 consensus_queue.cc:1048] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20251026 14:59:24.108079 22622 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:24.108824 22774 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.500s user 0.026s sys 0.044s
W20251026 14:59:24.110332 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:24.036310 24727 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.092868 24772 raft_consensus.cc:697] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:24.093428 24752 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:24.050803 24336 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.103775 24610 raft_consensus.cc:2749] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20251026 14:59:24.127429 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:24.110877 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:24.135949 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:24.054538 24587 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:24.134848 22774 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.526s user 0.032s sys 0.045s
I20251026 14:59:24.141160 22624 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:24.143420 22732 raft_consensus.cc:2468] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
W20251026 14:59:24.146071 24780 log.cc:927] Time spent T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.072s user 0.000s sys 0.001s
I20251026 14:59:24.146452 24784 raft_consensus.cc:2804] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:24.146876 24784 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:24.148864 24740 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:24.149418 24725 log.cc:927] Time spent T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 1.195s user 0.000s sys 0.008s
I20251026 14:59:24.149499 24736 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:24.150293 24115 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:24.151131 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1060) took 1514 ms. Trace:
I20251026 14:59:24.151499 22651 rpcz_store.cc:276] 1026 14:59:22.637071 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:22.637410 (+ 339us) service_pool.cc:225] Handling call
1026 14:59:22.637838 (+ 428us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:22.646306 (+ 8468us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:22.646360 (+ 54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:22.647071 (+ 711us) log.cc:844] Serialized 26 byte log entry
1026 14:59:22.805611 (+158540us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:22.805665 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:22.805757 (+ 92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:23.892428 (+1086671us) spinlock_profiling.cc:243] Waited 1.48 ms on lock 0x7b440002304c. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8cdfc0905 00007fd8cdfc1322 00007fd8d0d0a231 00007fd8d0d05b5a 00007fd8d0cf991e 00007fd8d0cf8c66 00007fd8d0cf8888 00007fd8d0d022c1 00007fd8d0d02279 00007fd8d0d02209
1026 14:59:24.150717 (+258289us) raft_consensus.cc:1685] finished
1026 14:59:24.150738 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:24.150951 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7946,"raft.queue_time_us":13899,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":124,"spinlock_wait_cycles":3399936,"thread_start_us":158006,"threads_started":1,"wal-append.queue_time_us":211941}
I20251026 14:59:24.147204 24784 raft_consensus.cc:3060] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:24.159994 24031 raft_consensus.cc:2804] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:24.160552 24031 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:24.160636 24775 log.cc:927] Time spent T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.107s user 0.000s sys 0.001s
I20251026 14:59:24.160962 24031 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:24.077498 24769 consensus_queue.cc:1048] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.084s
I20251026 14:59:24.161857 24743 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:24.162647 24587 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:24.143254 24418 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:24.163367 24418 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.160024 24786 raft_consensus.cc:2804] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:24.165930 24786 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:24.166288 24786 raft_consensus.cc:3060] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:24.142474 24752 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.160987 24772 consensus_queue.cc:237] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:24.168247 24752 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:24.162364 24743 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:24.109313 24788 consensus_queue.cc:1048] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.061s
I20251026 14:59:24.171868 24743 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:24.171994 24593 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:24.147339 24336 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:24.172709 24745 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:24.173235 24418 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:24.147681 24774 consensus_queue.cc:1048] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.174s
I20251026 14:59:24.174180 24031 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.156832 24385 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:24.182634 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 130305us
I20251026 14:59:24.185216 24031 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:24.189795 22697 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:24.194087 22697 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:24.194799 24752 raft_consensus.cc:3060] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:24.195207 24785 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:24.195741 22697 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:24.195777 24785 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:24.202772 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 635839us
I20251026 14:59:24.210847 22774 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:24.212195 24587 raft_consensus.cc:2749] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:24.209602 24783 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:24.222334 22729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 906) took 1065 ms. Trace:
I20251026 14:59:24.223470 24727 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:24.214771 23368 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.020s sys 0.052s
W20251026 14:59:24.225303 24742 consensus_meta.cc:287] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.643s user 0.012s sys 0.000s
I20251026 14:59:24.229700 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:24.232160 22773 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:24.230506 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:24.233120 22658 raft_consensus.cc:2468] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:24.236629 24593 raft_consensus.cc:2804] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:24.237105 24593 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:24.237306 24785 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:24.237586 24593 raft_consensus.cc:3060] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:24.238338 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:24.238587 24726 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.239023 22658 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 4.
I20251026 14:59:24.239696 22731 raft_consensus.cc:2393] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 6: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:24.240569 24726 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:24.241019 23370 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:24.241501 24745 raft_consensus.cc:2764] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20251026 14:59:24.242172 24785 raft_consensus.cc:2804] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:24.242604 24785 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:24.243048 22653 raft_consensus.cc:2393] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 6: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:24.243185 24785 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:24.225755 22729 rpcz_store.cc:276] 1026 14:59:23.157129 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:23.157248 (+ 119us) service_pool.cc:225] Handling call
1026 14:59:23.158516 (+ 1268us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:23.158932 (+ 416us) raft_consensus.cc:1534] Early marking committed up to index 3
1026 14:59:23.187064 (+ 28132us) log.cc:844] Serialized 14 byte log entry
1026 14:59:23.603154 (+416090us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:24.200599 (+597445us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x7b2c0000e678. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cfa5d9b1 00007fd8cfa6cfe2 00007fd8d1199701 00007fd8d115dfda 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22
1026 14:59:24.203319 (+ 2720us) log.cc:844] Serialized 26 byte log entry
1026 14:59:24.203524 (+ 205us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:24.203589 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:24.203686 (+ 97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:24.221988 (+ 18302us) raft_consensus.cc:1685] finished
1026 14:59:24.222112 (+ 124us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:24.222248 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21923,"prepare.queue_time_us":120618,"prepare.run_cpu_time_us":6536,"prepare.run_wall_time_us":50100,"raft.queue_time_us":65556,"raft.run_cpu_time_us":357,"raft.run_wall_time_us":27977,"spinlock_wait_cycles":39521536,"thread_start_us":384162,"threads_started":2,"wal-append.queue_time_us":419211}
I20251026 14:59:24.249979 22800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1068) took 1254 ms. Trace:
I20251026 14:59:24.250532 22800 rpcz_store.cc:276] 1026 14:59:22.995449 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:22.996480 (+ 1031us) service_pool.cc:225] Handling call
1026 14:59:23.012601 (+ 16121us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:23.901260 (+888659us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:24.249740 (+348480us) spinlock_profiling.cc:243] Waited 887 ms on lock 0x7b54003354d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:24.249922 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2039352576}
I20251026 14:59:24.244292 24752 raft_consensus.cc:2749] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251026 14:59:24.252046 23368 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.651s user 0.020s sys 0.053s
I20251026 14:59:24.264963 22653 raft_consensus.cc:1240] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:24.269121 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:24.270051 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 66062.9us
I20251026 14:59:24.270222 22658 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
W20251026 14:59:24.266186 24749 log.cc:927] Time spent T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.699s user 0.005s sys 0.000s
I20251026 14:59:24.272955 24354 raft_consensus.cc:2749] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251026 14:59:24.289934 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 94279.5us
I20251026 14:59:24.293444 22805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1071) took 1279 ms. Trace:
I20251026 14:59:24.293901 22805 rpcz_store.cc:276] 1026 14:59:23.013939 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:23.015167 (+ 1228us) service_pool.cc:225] Handling call
1026 14:59:23.029790 (+ 14623us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:23.094411 (+ 64621us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:23.094455 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:23.100987 (+ 6532us) log.cc:844] Serialized 26 byte log entry
1026 14:59:23.160416 (+ 59429us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:23.160448 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:23.160520 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:24.265633 (+1105113us) spinlock_profiling.cc:243] Waited 3.32 ms on lock 0x7b440002304c. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8cdfc0905 00007fd8cdfc1322 00007fd8d0d0a231 00007fd8d0d05b5a 00007fd8d0cf991e 00007fd8d0cf8c66 00007fd8d0cf8888 00007fd8d0d022c1 00007fd8d0d02279 00007fd8d0d02209
1026 14:59:24.293080 (+ 27447us) raft_consensus.cc:1685] finished
1026 14:59:24.293109 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:24.293378 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4729,"raft.queue_time_us":39980,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":81,"spinlock_wait_cycles":7630720,"thread_start_us":74635,"threads_started":2,"wal-append.queue_time_us":109847}
I20251026 14:59:24.253087 24783 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.260675 24742 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:59:24.303378 22730 raft_consensus.cc:3060] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:24.286695 24589 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.306056 24589 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:24.306367 24723 consensus_queue.cc:260] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.307528 24723 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:24.310567 24788 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:24.312052 24784 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.312750 22728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 918) took 1044 ms. Trace:
I20251026 14:59:24.311120 24788 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.313122 22728 rpcz_store.cc:276] 1026 14:59:23.268376 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:23.270327 (+ 1951us) service_pool.cc:225] Handling call
1026 14:59:23.559777 (+289450us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:23.559982 (+ 205us) raft_consensus.cc:1534] Early marking committed up to index 4
1026 14:59:23.587781 (+ 27799us) log.cc:844] Serialized 14 byte log entry
1026 14:59:24.271913 (+684132us) log.cc:844] Serialized 14 byte log entry
1026 14:59:24.312369 (+ 40456us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:24.312416 (+ 47us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:24.312477 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:24.312557 (+ 80us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:24.312694 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":524005,"prepare.queue_time_us":1141316,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":85,"raft.queue_time_us":40561,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":6757,"thread_start_us":183314,"threads_started":2,"wal-append.queue_time_us":435893}
I20251026 14:59:24.316658 22774 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:24.315704 22730 raft_consensus.cc:1275] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20251026 14:59:24.318109 22730 pending_rounds.cc:85] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e: Aborting all ops after (but not including) 1
I20251026 14:59:24.318605 22730 pending_rounds.cc:107] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20251026 14:59:24.318835 22730 raft_consensus.cc:2889] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251026 14:59:24.319581 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 56833.5us
I20251026 14:59:24.320187 22700 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:24.322901 22730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 916) took 1073 ms. Trace:
I20251026 14:59:24.323180 22730 rpcz_store.cc:276] 1026 14:59:23.249369 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:23.249798 (+ 429us) service_pool.cc:225] Handling call
1026 14:59:23.265811 (+ 16013us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:24.303088 (+1037277us) spinlock_profiling.cc:243] Waited 3.42 ms on lock 0x7b54001c91a0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d10d396d 00007fd8d10ba0f4 00007fd8d113c097 00007fd8d1154b87 00007fd8d115637e 00007fd8d11606d9 00007fd8d1160fa6 00007fd8d115d92f 00007fd8d115d3de 00007fd8d61059d9
1026 14:59:24.322705 (+ 19617us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:24.322856 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7860096}
I20251026 14:59:24.326298 22700 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:24.330574 22774 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:24.324723 24745 raft_consensus.cc:2804] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election won for term 5
W20251026 14:59:24.333047 24793 log.cc:927] Time spent T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.106s user 0.001s sys 0.000s
I20251026 14:59:24.333105 24745 raft_consensus.cc:697] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:24.325872 24752 raft_consensus.cc:2804] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:24.334182 24752 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:24.334219 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 905) took 1177 ms. Trace:
I20251026 14:59:24.334686 24752 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:24.334422 24745 consensus_queue.cc:237] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.334800 22726 rpcz_store.cc:276] 1026 14:59:23.156776 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:23.158407 (+ 1631us) service_pool.cc:225] Handling call
1026 14:59:24.333985 (+1175578us) spinlock_profiling.cc:243] Waited 644 ms on lock 0x7b54000b90d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:24.334134 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1481717504}
W20251026 14:59:24.325124 24730 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20251026 14:59:24.317834 24788 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:24.347175 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:24.348553 22726 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:24.349646 24635 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:24.349699 22725 raft_consensus.cc:1275] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20251026 14:59:24.338826 24735 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:24.350471 24635 raft_consensus.cc:2804] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:24.350956 24635 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:24.350878 24735 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.351446 24635 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:24.350080 22728 raft_consensus.cc:1275] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:24.361888 22730 raft_consensus.cc:1275] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:24.331555 24784 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:24.334860 24783 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:24.365733 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:24.366707 22728 raft_consensus.cc:3060] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:24.329917 24786 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.371937 22774 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:24.376154 24785 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:24.377635 22697 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:24.378700 24742 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:24.366650 24587 raft_consensus.cc:493] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:24.383930 22725 raft_consensus.cc:1275] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20251026 14:59:24.384347 22725 pending_rounds.cc:85] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e: Aborting all ops after (but not including) 2
I20251026 14:59:24.384574 22725 pending_rounds.cc:107] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20251026 14:59:24.384806 22725 raft_consensus.cc:2889] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:24.391862 24745 raft_consensus.cc:2804] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:24.361245 24788 raft_consensus.cc:2749] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:24.396263 24587 raft_consensus.cc:515] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:24.398008 24785 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:24.399159 24587 leader_election.cc:290] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:24.392977 24774 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:24.400658 24774 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.403626 24774 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:24.405895 24786 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:24.407900 24752 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.420862 22728 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
W20251026 14:59:24.409977 22733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 907) took 1248 ms (client timeout 1628 ms). Trace:
W20251026 14:59:24.422897 22733 rpcz_store.cc:269] 1026 14:59:23.161969 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:23.162097 (+ 128us) service_pool.cc:225] Handling call
1026 14:59:24.409683 (+1247586us) spinlock_profiling.cc:243] Waited 390 ms on lock 0x7b540028b754. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:24.409896 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":897570048}
I20251026 14:59:24.428721 24735 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:24.431324 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:24.426198 24786 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:24.392566 24745 raft_consensus.cc:697] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:24.425657 22774 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:24.442354 24786 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.443500 24788 consensus_queue.cc:1059] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:59:24.444226 24788 consensus_queue.cc:1243] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer df3a0d1619ee48e8ab68ee0543a72a29 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251026 14:59:24.444532 24786 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:24.444633 24774 raft_consensus.cc:3055] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:24.445029 22774 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:24.450137 22774 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:24.426959 24610 raft_consensus.cc:2749] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 14:59:24.445102 24774 raft_consensus.cc:740] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:24.443492 24625 raft_consensus.cc:2804] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:24.456827 24740 raft_consensus.cc:2804] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:24.456686 24774 consensus_queue.cc:260] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.457422 24740 raft_consensus.cc:697] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:24.457755 24774 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:24.458115 24625 raft_consensus.cc:697] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:24.458328 24740 consensus_queue.cc:237] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.459059 24625 consensus_queue.cc:237] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.449733 24742 raft_consensus.cc:2749] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251026 14:59:24.456687 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 315354us
I20251026 14:59:24.463536 22622 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:24.464610 24748 raft_consensus.cc:2804] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:24.464860 24527 consensus_queue.cc:1059] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:59:24.465087 24748 raft_consensus.cc:697] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:24.443689 24745 consensus_queue.cc:237] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:24.468047 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 90688.5us
I20251026 14:59:24.470638 22733 raft_consensus.cc:2468] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:24.471423 22554 catalog_manager.cc:5649] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "248493ec312e4398b828c6cb00e5c4b1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 14:59:24.472011 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:24.472823 22658 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 6
I20251026 14:59:24.473502 24527 raft_consensus.cc:3055] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:24.473953 24527 raft_consensus.cc:740] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:24.474829 24527 consensus_queue.cc:260] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.475775 22801 raft_consensus.cc:1275] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:24.475960 24527 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:24.478510 24752 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:24.479602 24748 consensus_queue.cc:237] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.479571 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:59:24.470124 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:24.486323 22725 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 2.
I20251026 14:59:24.491369 22774 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:24.502245 22554 catalog_manager.cc:5649] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "248493ec312e4398b828c6cb00e5c4b1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 14:59:24.503973 22554 catalog_manager.cc:5649] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: term changed from 3 to 4, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3), no longer has a pending config: VOTER df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1), VOTER 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2), VOTER 0d684445287b450a8f738b6ad4b59821 (127.21.2.3), VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4). New cstate: current_term: 4 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:59:24.509016 22658 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 6.
I20251026 14:59:24.520859 22554 catalog_manager.cc:5649] T 5373ef47363d446fbf9d697ed507e754 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "248493ec312e4398b828c6cb00e5c4b1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 14:59:24.506949 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:24.524125 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:24.524685 22774 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:24.525317 22725 raft_consensus.cc:2393] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 6: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:24.528995 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:24.529681 22808 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:24.468478 24635 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.523515 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:24.505107 24742 raft_consensus.cc:2804] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:24.531905 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:24.532675 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:24.534710 22807 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:24.538384 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:24.539019 22803 raft_consensus.cc:3060] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 4
I20251026 14:59:24.542569 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:24.542959 22801 raft_consensus.cc:1275] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251026 14:59:24.543213 22804 raft_consensus.cc:3060] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:24.547468 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:24.548098 22808 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:24.550979 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:24.533156 24332 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:24.555509 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:24.518529 24786 consensus_queue.cc:1048] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.027s
I20251026 14:59:24.558014 23370 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:24.558866 22774 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:24.555300 24737 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:24.559574 24787 raft_consensus.cc:2804] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:24.563107 22807 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:24.565505 24737 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.566552 24787 raft_consensus.cc:697] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:24.566882 24635 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:24.567914 24770 raft_consensus.cc:2804] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:24.567606 24787 consensus_queue.cc:237] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.568343 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:24.569518 22725 raft_consensus.cc:2393] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 5: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:24.569875 24781 raft_consensus.cc:2749] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:24.570575 24593 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.573472 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:24.575554 22805 raft_consensus.cc:1240] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:24.577184 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:24.578378 22726 raft_consensus.cc:2393] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 6: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:24.544903 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
W20251026 14:59:24.579885 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 82715.2us
I20251026 14:59:24.581462 22774 leader_election.cc:400] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6.
I20251026 14:59:24.588836 22774 leader_election.cc:403] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20251026 14:59:24.592788 22803 raft_consensus.cc:2468] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:24.554101 24742 raft_consensus.cc:697] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:24.593873 22799 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:24.593003 22804 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:24.591131 24781 raft_consensus.cc:2749] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6.
I20251026 14:59:24.586712 24332 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.568483 24770 raft_consensus.cc:697] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 LEADER]: Becoming Leader. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:24.605866 22803 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e"
dest_uuid: "0d684445287b450a8f738b6ad4b59821"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:44012
I20251026 14:59:24.606554 22803 raft_consensus.cc:606] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:24.606993 22803 raft_consensus.cc:614] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251026 14:59:24.607971 22774 leader_election.cc:400] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6.
I20251026 14:59:24.608289 22774 leader_election.cc:403] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20251026 14:59:24.607095 22808 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:24.613474 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:24.619145 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:24.629112 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:24.633754 22730 raft_consensus.cc:3060] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:24.617671 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:24.613925 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:24.616693 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:24.634544 22726 raft_consensus.cc:3060] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:24.634902 22727 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:24.618515 22733 raft_consensus.cc:1275] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:24.617008 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:24.637912 22622 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:24.617365 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:24.638764 22725 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:24.638972 22731 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:24.643177 22800 raft_consensus.cc:2410] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251026 14:59:24.637763 22697 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:24.645875 24786 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:24.646404 24786 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.647035 22697 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:24.648486 24786 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:24.648792 22697 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:24.649451 24610 raft_consensus.cc:2749] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:24.649509 22803 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "627ebd70757a47b18827c38f071b5261"
dest_uuid: "0d684445287b450a8f738b6ad4b59821"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:44012
I20251026 14:59:24.650112 22622 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:24.650259 22803 raft_consensus.cc:606] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:24.650636 22803 raft_consensus.cc:614] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251026 14:59:24.650713 24353 raft_consensus.cc:2804] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:24.651153 24353 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:24.651476 24353 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:24.651928 24784 raft_consensus.cc:2804] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:24.651876 24734 consensus_queue.cc:1048] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:59:24.652546 24784 raft_consensus.cc:697] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:24.653362 22725 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:24.651417 22622 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:24.655786 22725 raft_consensus.cc:1240] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20251026 14:59:24.655750 22730 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:24.607215 24593 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:24.610244 24781 raft_consensus.cc:2749] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6.
I20251026 14:59:24.664891 22726 raft_consensus.cc:2468] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:24.669804 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:24.619874 24332 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:24.666819 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:24.653949 24784 consensus_queue.cc:237] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.672652 24769 raft_consensus.cc:2804] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:24.673420 24769 raft_consensus.cc:697] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:24.673540 24786 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:24.676381 22800 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb"
dest_uuid: "0d684445287b450a8f738b6ad4b59821"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:44012
I20251026 14:59:24.667946 22622 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:24.677225 22800 raft_consensus.cc:606] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:24.678289 22800 raft_consensus.cc:614] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251026 14:59:24.650290 24610 raft_consensus.cc:2749] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:24.674057 24786 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.680102 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:24.674711 24635 raft_consensus.cc:2749] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:24.688210 22730 raft_consensus.cc:1240] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
W20251026 14:59:24.688913 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 98717.6us
W20251026 14:59:24.692356 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:24.693747 22622 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:24.695904 22622 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:24.696698 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 47985.6us
I20251026 14:59:24.697559 22553 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38884 (request call id 156) took 10718 ms. Trace:
I20251026 14:59:24.678653 24737 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:24.621049 24742 consensus_queue.cc:237] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.699261 24735 raft_consensus.cc:2804] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:24.698020 24032 raft_consensus.cc:2804] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:24.704732 24332 consensus_queue.cc:1059] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.002s
I20251026 14:59:24.674320 24769 consensus_queue.cc:237] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.707098 24593 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:24.678602 24770 consensus_queue.cc:237] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:24.713491 22807 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb"
dest_uuid: "0d684445287b450a8f738b6ad4b59821"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:44012
I20251026 14:59:24.719940 22807 raft_consensus.cc:606] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:24.720394 22807 raft_consensus.cc:614] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251026 14:59:24.704720 24781 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:24.704789 24735 raft_consensus.cc:697] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:24.728673 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:24.729494 22697 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:24.735760 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:24.739673 24032 raft_consensus.cc:697] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:24.740083 22728 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 6
I20251026 14:59:24.740947 24735 consensus_queue.cc:237] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.741927 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:24.742918 22802 raft_consensus.cc:2410] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251026 14:59:24.744432 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:24.744550 24781 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.746907 22658 raft_consensus.cc:1275] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:24.747030 24781 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:24.748082 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:24.720146 24745 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:24.749694 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:24.749745 24745 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.752148 24745 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:24.752483 24786 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:24.697960 22553 rpcz_store.cc:276] 1026 14:59:13.969688 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:13.971707 (+ 2019us) service_pool.cc:225] Handling call
1026 14:59:24.685109 (+10713402us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 14:59:22.282921 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 14:59:22.283850 (+ 929us) write_op.cc:432] Acquiring schema lock in shared mode
1026 14:59:22.283913 (+ 63us) write_op.cc:435] Acquired schema lock
1026 14:59:22.283946 (+ 33us) tablet.cc:660] Decoding operations
1026 14:59:22.325598 (+ 41652us) write_op.cc:620] Acquiring the partition lock for write op
1026 14:59:22.326633 (+ 1035us) write_op.cc:641] Partition lock acquired for write op
1026 14:59:22.326675 (+ 42us) tablet.cc:683] Acquiring locks for 4 operations
1026 14:59:22.327256 (+ 581us) tablet.cc:699] Row locks acquired
1026 14:59:22.327289 (+ 33us) write_op.cc:260] PREPARE: finished
1026 14:59:22.327477 (+ 188us) write_op.cc:270] Start()
1026 14:59:22.327646 (+ 169us) write_op.cc:276] Timestamp: P: 1761490762327411 usec, L: 0
1026 14:59:22.327686 (+ 40us) op_driver.cc:348] REPLICATION: starting
1026 14:59:22.328588 (+ 902us) log.cc:844] Serialized 1665 byte log entry
1026 14:59:22.774751 (+446163us) op_driver.cc:464] REPLICATION: finished
1026 14:59:22.914665 (+139914us) write_op.cc:301] APPLY: starting
1026 14:59:22.919917 (+ 5252us) tablet.cc:1365] starting BulkCheckPresence
1026 14:59:22.950692 (+ 30775us) tablet.cc:1368] finished BulkCheckPresence
1026 14:59:22.985500 (+ 34808us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 14:59:23.108196 (+122696us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 14:59:23.108486 (+ 290us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
1026 14:59:23.108549 (+ 63us) write_op.cc:312] APPLY: finished
1026 14:59:23.108845 (+ 296us) log.cc:844] Serialized 40 byte log entry
1026 14:59:23.109060 (+ 215us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:23.109164 (+ 104us) write_op.cc:454] Released schema lock
1026 14:59:23.109233 (+ 69us) write_op.cc:341] FINISH: Updating metrics
1026 14:59:23.109709 (+ 476us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:23.109746 (+ 37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":907360,"reported_tablets":36,"spinlock_wait_cycles":131712,"child_traces":[["op",{"apply.queue_time_us":124527,"apply.run_cpu_time_us":3773,"apply.run_wall_time_us":202700,"num_ops":4,"prepare.queue_time_us":505713,"prepare.run_cpu_time_us":5662,"prepare.run_wall_time_us":72192,"raft.queue_time_us":384196,"raft.run_cpu_time_us":9553,"raft.run_wall_time_us":104003,"replication_time_us":432409,"thread_start_us":530697,"threads_started":4,"wal-append.queue_time_us":21598,"wal-append.run_cpu_time_us":67351,"wal-append.run_wall_time_us":1784052}]]}
I20251026 14:59:24.761651 22728 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:24.764238 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:24.765305 22652 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:24.768942 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:24.769881 22730 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:24.773559 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:24.775998 22773 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:24.720530 24748 raft_consensus.cc:2804] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:24.779022 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:24.781669 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:24.786985 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:24.787922 22726 raft_consensus.cc:3060] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:24.792340 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:24.793365 22658 raft_consensus.cc:2410] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20251026 14:59:24.793395 22654 raft_consensus.cc:1275] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251026 14:59:24.774533 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:24.798811 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:24.801738 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:24.802254 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:24.802876 22657 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:24.802552 22656 raft_consensus.cc:2393] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 5: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
W20251026 14:59:24.805312 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 75862.5us
I20251026 14:59:24.770754 24782 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:24.815289 22697 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:24.766479 24752 raft_consensus.cc:2749] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:24.820609 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:24.822324 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:24.820688 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:24.824479 22659 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:24.827117 22658 raft_consensus.cc:1275] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20251026 14:59:24.827481 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:24.802536 24745 raft_consensus.cc:2804] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:24.829846 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:24.759160 24593 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.830236 22657 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:24.830631 22728 raft_consensus.cc:2468] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:24.831087 22773 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:24.831593 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:24.802700 24748 raft_consensus.cc:697] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:24.835203 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:24.835777 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:24.836354 22730 raft_consensus.cc:2410] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251026 14:59:24.836593 22651 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:24.820986 24787 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:24.845588 22774 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:24.846516 22700 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:24.846925 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:24.847151 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:24.849541 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:24.850028 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:24.850746 22774 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:24.851138 22773 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:24.848887 22726 raft_consensus.cc:2468] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:24.808004 24625 consensus_queue.cc:1059] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.031s
I20251026 14:59:24.852905 22657 raft_consensus.cc:1275] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:24.853111 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:24.923892 22650 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:24.822346 24635 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:24.856850 22774 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:24.934996 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 99847.2us
I20251026 14:59:24.835307 24737 raft_consensus.cc:2764] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20251026 14:59:24.740993 24032 consensus_queue.cc:237] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.824301 24782 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.830209 24745 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:24.807127 24353 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:24.905983 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:24.928443 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:24.821380 24759 consensus_queue.cc:1048] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20251026 14:59:24.815884 24781 consensus_queue.cc:1059] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20251026 14:59:24.858814 24727 raft_consensus.cc:2804] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:24.979982 22650 raft_consensus.cc:2468] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:24.858572 24741 raft_consensus.cc:2749] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:24.915454 24730 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:24.846762 24752 consensus_queue.cc:1048] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.101s
I20251026 14:59:24.913614 24788 raft_consensus.cc:2749] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:24.929747 24736 raft_consensus.cc:2804] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:24.975847 22659 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:24.833606 24769 consensus_queue.cc:1048] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.110s
I20251026 14:59:24.961133 22800 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:24.951377 24806 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:24.962584 22658 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570"
dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:46598
I20251026 14:59:25.047987 22658 raft_consensus.cc:606] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:25.048468 22658 raft_consensus.cc:614] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251026 14:59:24.854737 24115 raft_consensus.cc:2804] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:24.855325 24774 consensus_queue.cc:1048] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.095s
I20251026 14:59:24.980623 24727 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:24.965116 24745 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:24.974277 22697 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:24.990278 24527 consensus_meta.cc:287] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.514s user 0.010s sys 0.006s
I20251026 14:59:25.045065 24736 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
W20251026 14:59:25.050652 24791 log.cc:927] Time spent T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
W20251026 14:59:25.075949 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 280298us
I20251026 14:59:24.943760 24635 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:25.058168 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 212638us
I20251026 14:59:24.934788 24787 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:25.040413 24816 log.cc:927] Time spent T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.087s user 0.000s sys 0.001s
W20251026 14:59:25.057596 24827 log.cc:927] Time spent T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1: Append to log took a long time: real 0.098s user 0.001s sys 0.000s
I20251026 14:59:25.085322 24786 consensus_queue.cc:1048] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20251026 14:59:25.085907 24822 log.cc:927] Time spent T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.145s user 0.001s sys 0.000s
W20251026 14:59:25.056798 24828 log.cc:927] Time spent T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20251026 14:59:25.047613 24741 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:25.087174 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 113034us
I20251026 14:59:25.087535 24786 consensus_queue.cc:1048] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:59:25.087126 24741 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:25.088683 24763 raft_consensus.cc:2749] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251026 14:59:25.080842 24734 consensus_queue.cc:1048] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251026 14:59:25.089430 24766 consensus_queue.cc:1048] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:59:25.090262 24741 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:25.090560 24763 consensus_queue.cc:1048] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:59:25.091836 22624 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:25.092628 24843 raft_consensus.cc:2749] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:25.097756 22624 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:25.059144 24784 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:25.034780 24730 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:25.071482 24727 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:25.089145 22697 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:25.066236 24737 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:25.113024 22700 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:25.104732 22773 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:25.070477 24115 raft_consensus.cc:697] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:25.109558 22651 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
I20251026 14:59:25.074477 24736 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:25.117147 22697 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:25.112438 24819 log.cc:927] Time spent T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.334s user 0.001s sys 0.000s
I20251026 14:59:25.040789 24748 consensus_queue.cc:237] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:25.105868 24843 raft_consensus.cc:2749] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:25.058060 24806 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:25.147176 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 290403us
I20251026 14:59:25.121049 24763 raft_consensus.cc:2749] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:25.150067 24115 consensus_queue.cc:237] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
W20251026 14:59:25.150851 24818 log.cc:927] Time spent T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.223s user 0.001s sys 0.000s
I20251026 14:59:25.153213 22774 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:25.120757 24741 raft_consensus.cc:2749] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:25.159569 22774 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:25.130486 24625 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:25.153427 24610 raft_consensus.cc:2804] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:25.161437 22774 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:25.162097 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:25.166623 24834 raft_consensus.cc:2804] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:25.167384 24834 raft_consensus.cc:697] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:25.168282 24625 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:25.168965 22774 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:25.173743 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:25.178496 24635 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:25.107825 24784 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:25.187719 24784 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:25.190840 24353 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:25.192277 24824 log.cc:927] Time spent T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20251026 14:59:25.148510 24593 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:25.192162 24837 raft_consensus.cc:2804] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:25.119483 24737 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:25.171080 24625 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:25.157361 24766 raft_consensus.cc:2749] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:25.155319 24752 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:25.168642 24834 consensus_queue.cc:237] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:25.215667 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:25.226080 22730 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:25.211091 24593 raft_consensus.cc:2749] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:25.228142 22773 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:25.276357 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 107473us
I20251026 14:59:25.211898 24741 raft_consensus.cc:2749] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:25.278403 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251026 14:59:25.282696 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:59:25.212427 24837 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:25.288070 24846 consensus_queue.cc:1048] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20251026 14:59:25.288470 22774 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:25.232008 24752 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:25.294471 24752 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:25.275517 24787 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:25.309625 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 81606.8us
W20251026 14:59:25.303500 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 186453us
I20251026 14:59:25.172544 24730 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:59:25.182612 24782 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:25.312908 22697 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:25.297894 24774 raft_consensus.cc:2749] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:25.287649 24837 raft_consensus.cc:3060] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
W20251026 14:59:25.324470 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 170243us
I20251026 14:59:25.280827 24353 consensus_queue.cc:1048] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.032s
I20251026 14:59:25.322852 22697 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:25.306383 24610 raft_consensus.cc:697] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:25.338941 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:25.339826 22725 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:25.307618 24032 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:25.344975 22622 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:25.324595 24730 raft_consensus.cc:2749] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:25.329841 24836 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:25.325345 24835 raft_consensus.cc:2804] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:25.349516 22773 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:25.351722 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 63352.2us
I20251026 14:59:25.353528 22655 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
W20251026 14:59:25.378499 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 55766.5us
I20251026 14:59:25.347775 23401 raft_consensus.cc:1275] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251026 14:59:25.320889 24752 consensus_queue.cc:1048] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20251026 14:59:25.344662 24763 raft_consensus.cc:2749] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:25.343531 24032 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:25.380048 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:25.417330 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:25.417805 22805 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:25.325591 22624 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:25.382432 22774 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:25.383462 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:25.349436 24806 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:25.368220 24840 raft_consensus.cc:2804] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20251026 14:59:25.384069 24839 log.cc:927] Time spent T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
I20251026 14:59:25.425223 22808 raft_consensus.cc:1275] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20251026 14:59:25.426615 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 77173.2us
I20251026 14:59:25.386049 24835 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:25.431828 24846 raft_consensus.cc:2804] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:25.383584 24847 raft_consensus.cc:2749] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:25.433065 22725 raft_consensus.cc:1275] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:25.434245 22733 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:25.434983 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 52644.9us
W20251026 14:59:25.432555 24838 log.cc:927] Time spent T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.128s user 0.000s sys 0.001s
I20251026 14:59:25.425834 24840 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:25.440559 24032 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:25.443214 24610 consensus_queue.cc:237] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:25.446154 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 120693us
I20251026 14:59:25.495177 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:25.495465 22624 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:25.496094 22659 raft_consensus.cc:2393] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:25.446700 24353 raft_consensus.cc:2749] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:25.446945 22726 raft_consensus.cc:1240] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20251026 14:59:25.369952 24836 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:25.422295 24804 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:25.439361 24835 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:25.473673 22805 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:25.420310 24031 raft_consensus.cc:3055] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:25.490274 22725 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:25.482878 24737 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:25.493997 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:25.549460 22651 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:25.501022 22700 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:25.550745 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 49807.2us
I20251026 14:59:25.501264 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:25.438310 24846 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:25.509805 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:25.511575 22799 raft_consensus.cc:1240] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:25.490406 24741 raft_consensus.cc:2749] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:25.512282 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:25.565514 22805 raft_consensus.cc:2368] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 2. Current term is 3.
W20251026 14:59:25.491020 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 71340.1us
I20251026 14:59:25.514453 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:25.515275 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:25.491660 23401 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:25.517563 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:25.519179 24836 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:25.520551 24742 raft_consensus.cc:3055] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:25.521633 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:25.511935 24031 raft_consensus.cc:740] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:25.526167 22624 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
W20251026 14:59:25.587811 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 61721.2us
W20251026 14:59:25.528960 24848 log.cc:927] Time spent T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.160s user 0.000s sys 0.005s
I20251026 14:59:25.490245 24774 consensus_queue.cc:1048] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20251026 14:59:25.490927 24740 consensus_queue.cc:1048] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
W20251026 14:59:25.530392 24833 log.cc:927] Time spent T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.228s user 0.001s sys 0.006s
I20251026 14:59:25.594650 22656 raft_consensus.cc:2393] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 6: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:25.595935 22654 raft_consensus.cc:1275] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:25.531057 22733 raft_consensus.cc:1275] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:25.532476 22806 raft_consensus.cc:1275] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:25.588936 22624 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:25.533021 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:25.534253 22622 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:25.534946 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:25.535207 24830 consensus_queue.cc:1048] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:59:25.537017 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:25.537159 24763 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:25.537602 24806 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:25.538268 24850 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:25.564982 24856 consensus_queue.cc:1059] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.043s
I20251026 14:59:25.608152 24850 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:25.610170 22656 raft_consensus.cc:1240] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20251026 14:59:25.610369 24850 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:25.539198 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:25.538659 24766 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:25.538720 24730 raft_consensus.cc:493] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:25.611223 24766 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:25.611683 24730 raft_consensus.cc:515] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:25.612071 22656 raft_consensus.cc:1275] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:25.613377 24766 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:25.554840 22807 raft_consensus.cc:1275] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:25.492079 24840 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:25.450810 24727 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:25.618031 22728 raft_consensus.cc:2368] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7.
I20251026 14:59:25.568684 22658 raft_consensus.cc:2393] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:25.573669 22726 raft_consensus.cc:3060] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:25.613718 24730 leader_election.cc:290] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:25.578081 22804 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:25.577550 22725 raft_consensus.cc:1275] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:25.639439 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:25.581363 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:25.584714 22697 leader_election.cc:400] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 2. Current term is 3.
I20251026 14:59:25.591614 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:25.641695 22651 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:25.585760 24742 raft_consensus.cc:740] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:25.598986 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:25.647368 22799 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:25.603883 22659 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:25.582911 24737 consensus_queue.cc:1059] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
I20251026 14:59:25.565389 24846 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:25.650805 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:25.606739 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 72556.6us
I20251026 14:59:25.606869 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:25.651880 22806 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:25.607604 24842 raft_consensus.cc:2804] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:25.608919 24741 consensus_queue.cc:1048] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20251026 14:59:25.613502 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:25.614761 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:25.525254 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:25.654277 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:25.505992 24032 raft_consensus.cc:2804] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:25.619133 24727 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:25.513638 24804 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:25.606890 24854 raft_consensus.cc:2749] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251026 14:59:25.628319 24745 consensus_meta.cc:287] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.549s user 0.006s sys 0.006s
I20251026 14:59:25.628942 24788 raft_consensus.cc:3055] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:25.635576 24806 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:25.664919 24782 consensus_queue.cc:1048] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251026 14:59:25.633816 24763 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:25.666929 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:25.667122 24806 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:25.667665 24763 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:25.649765 24031 consensus_queue.cc:260] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:25.668231 22727 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:25.668315 24842 raft_consensus.cc:697] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:25.668601 24031 raft_consensus.cc:3060] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:25.669354 24842 consensus_queue.cc:237] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:25.672266 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:25.673519 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:25.674091 22803 raft_consensus.cc:2468] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
W20251026 14:59:25.674722 24736 consensus_meta.cc:287] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.527s user 0.011s sys 0.000s
I20251026 14:59:25.674978 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:25.675182 24736 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:25.668259 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:25.677479 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:25.678453 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:25.678671 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:25.679407 22808 raft_consensus.cc:2393] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 6: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:25.679630 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:25.680125 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:25.680770 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:25.681727 24844 consensus_queue.cc:1048] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20251026 14:59:25.682925 22799 raft_consensus.cc:2393] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 6: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:25.677517 22727 raft_consensus.cc:1275] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251026 14:59:25.676069 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:25.684238 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:25.659308 24032 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:25.643469 22731 raft_consensus.cc:1275] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:25.689653 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:25.578179 24838 log.cc:927] Time spent T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.080s user 0.000s sys 0.001s
I20251026 14:59:25.615631 24841 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:25.601200 24836 raft_consensus.cc:2804] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:25.686390 24782 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:25.691834 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:25.643942 22697 leader_election.cc:403] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20251026 14:59:25.645087 22732 raft_consensus.cc:1275] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251026 14:59:25.565264 24752 raft_consensus.cc:2804] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:25.655426 22774 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:25.748250 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:25.662911 24788 raft_consensus.cc:740] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:25.687631 22653 raft_consensus.cc:1240] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:25.697871 22804 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:25.631680 22726 raft_consensus.cc:1275] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20251026 14:59:25.749311 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 93991.7us
I20251026 14:59:25.788910 22803 raft_consensus.cc:1275] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251026 14:59:25.686044 24332 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:25.691365 24032 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:25.797425 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:25.844058 22732 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:25.690389 22733 raft_consensus.cc:2468] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:25.748435 24836 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:25.683450 24745 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:25.855764 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 280789us
I20251026 14:59:25.845837 22624 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:25.855698 22733 raft_consensus.cc:1275] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20251026 14:59:25.788067 23369 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.546s (SENT) [suppressed 12 similar messages]
I20251026 14:59:25.863601 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:25.866806 22624 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:25.846828 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 58026.1us
I20251026 14:59:25.792677 23401 raft_consensus.cc:2410] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20251026 14:59:25.792357 24844 raft_consensus.cc:2749] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:25.889390 22801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1115) took 1155 ms. Trace:
W20251026 14:59:25.889969 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 253249us
I20251026 14:59:25.785823 24782 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:25.922596 22807 raft_consensus.cc:2368] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 6. Current term is 7.
I20251026 14:59:25.855192 24727 raft_consensus.cc:2749] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 2. Current term is 3.
I20251026 14:59:25.884964 24836 raft_consensus.cc:3060] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:25.906503 24804 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:25.888506 22651 raft_consensus.cc:1275] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20251026 14:59:25.790288 24736 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:25.793751 24752 raft_consensus.cc:697] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 8 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:25.881527 24845 raft_consensus.cc:2749] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:25.901882 24788 consensus_queue.cc:260] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:25.971875 22733 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:25.749827 24840 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:25.792711 24847 consensus_queue.cc:1048] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.132s
I20251026 14:59:25.918445 22801 rpcz_store.cc:276] 1026 14:59:24.734027 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:24.734135 (+ 108us) service_pool.cc:225] Handling call
1026 14:59:24.743875 (+ 9740us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:24.778488 (+ 34613us) raft_consensus.cc:1534] Early marking committed up to index 3
1026 14:59:24.833083 (+ 54595us) log.cc:844] Serialized 14 byte log entry
1026 14:59:25.157382 (+324299us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:25.163433 (+ 6051us) log.cc:844] Serialized 26 byte log entry
1026 14:59:25.163582 (+ 149us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:25.163615 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:25.163691 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:25.289751 (+126060us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b480027da58. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d606b971 00007fd8d6062a37 00007fd8d60628ca 00007fd8d6159a0f 00007fd8d61596d7 00007fd8d616dcfc 00007fd8d616dca8 00007fd8d616dc28 00007fd8d616dbe3 00007fd8d616ce8c 00007fd8d50aa326
1026 14:59:25.846958 (+557207us) raft_consensus.cc:1685] finished
1026 14:59:25.889162 (+ 42204us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:25.889326 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":96927,"prepare.queue_time_us":50925,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":4258,"raft.queue_time_us":62756,"raft.run_cpu_time_us":3537,"raft.run_wall_time_us":516996,"spinlock_wait_cycles":31285632,"thread_start_us":215799,"threads_started":1,"wal-append.queue_time_us":383791}
I20251026 14:59:25.691830 24742 consensus_queue.cc:260] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.006811 22730 raft_consensus.cc:1240] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20251026 14:59:25.843851 24332 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:25.919186 24741 consensus_queue.cc:1048] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
I20251026 14:59:26.066852 22733 raft_consensus.cc:1275] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:59:26.083292 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:26.056934 22625 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.015s sys 0.053s
W20251026 14:59:26.085983 22625 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.584s user 0.015s sys 0.053s
W20251026 14:59:26.087518 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:26.060956 24854 raft_consensus.cc:2749] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:26.116256 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:26.117105 22801 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 7
I20251026 14:59:26.056624 24853 consensus_queue.cc:1048] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20251026 14:59:26.062844 24625 consensus_queue.cc:1059] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.046s
I20251026 14:59:26.060724 24527 consensus_queue.cc:1048] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.051s
I20251026 14:59:26.080554 24856 consensus_queue.cc:1059] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20251026 14:59:26.146723 23404 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
W20251026 14:59:26.148978 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 360927us
W20251026 14:59:26.152058 22802 consensus_meta.cc:287] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.584s user 0.012s sys 0.000s
I20251026 14:59:26.152667 22802 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:26.154012 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:26.165086 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:26.151518 24752 consensus_queue.cc:237] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.178772 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:26.175169 24823 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.790s user 0.022s sys 0.018s
I20251026 14:59:26.149173 24741 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:26.195716 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:25.748314 24841 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:26.203362 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:26.204509 22650 raft_consensus.cc:2393] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 5: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
W20251026 14:59:26.192199 24823 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.809s user 0.023s sys 0.018s
I20251026 14:59:26.193804 24846 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:26.243386 24868 log.cc:927] Time spent T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20251026 14:59:26.244359 24846 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:26.247025 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:26.258016 22650 raft_consensus.cc:1275] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20251026 14:59:26.262985 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 396264us
W20251026 14:59:26.260663 24031 consensus_meta.cc:287] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.591s user 0.004s sys 0.008s
W20251026 14:59:26.265791 24863 log.cc:927] Time spent T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.151s user 0.001s sys 0.000s
I20251026 14:59:26.304595 22650 raft_consensus.cc:1240] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 6. Current term is 7. Ops: []
W20251026 14:59:26.305500 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:26.308600 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:26.310411 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:26.311442 22651 raft_consensus.cc:2393] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
W20251026 14:59:26.345772 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 79228.5us
I20251026 14:59:26.346828 22624 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:26.348491 22624 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:26.354422 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:26.206826 24741 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.355291 22658 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:26.309592 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:26.354130 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:26.362092 22653 raft_consensus.cc:2393] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 6: Already voted for candidate 248493ec312e4398b828c6cb00e5c4b1 in this term.
I20251026 14:59:26.307845 24782 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:26.358906 24862 raft_consensus.cc:2804] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251026 14:59:26.399573 24869 log.cc:927] Time spent T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.203s user 0.000s sys 0.000s
I20251026 14:59:26.299764 24032 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.354146 24031 raft_consensus.cc:2804] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:26.312748 24840 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:26.404367 24876 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:26.405539 23399 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43606 (request call id 53) took 1888 ms (client timeout 1762 ms). Trace:
I20251026 14:59:26.403483 24741 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:26.362367 24748 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:26.395128 24332 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:26.480376 24837 consensus_meta.cc:287] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 1.142s user 0.011s sys 0.003s
I20251026 14:59:26.463747 24862 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:26.525342 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 174733us
I20251026 14:59:26.525540 22652 raft_consensus.cc:1275] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20251026 14:59:26.527166 22657 raft_consensus.cc:1240] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:26.480835 24031 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:26.536183 22658 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
W20251026 14:59:26.541172 24870 log.cc:927] Time spent T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.335s user 0.000s sys 0.000s
W20251026 14:59:26.542686 24883 log.cc:927] Time spent T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d: Append to log took a long time: real 0.138s user 0.000s sys 0.001s
I20251026 14:59:26.546636 22734 raft_consensus.cc:1275] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251026 14:59:26.532140 24782 consensus_queue.cc:1048] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20251026 14:59:26.538611 24862 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:26.550302 22799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1096) took 1020 ms. Trace:
W20251026 14:59:26.542761 24867 log.cc:927] Time spent T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.281s user 0.000s sys 0.000s
I20251026 14:59:26.551003 22804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1097) took 1020 ms. Trace:
I20251026 14:59:26.550814 22799 rpcz_store.cc:276] 1026 14:59:25.530243 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:25.785580 (+255337us) service_pool.cc:225] Handling call
1026 14:59:25.789903 (+ 4323us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:25.854403 (+ 64500us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:25.854433 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:25.880911 (+ 26478us) log.cc:844] Serialized 52 byte log entry
1026 14:59:26.122644 (+241733us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:26.306296 (+183652us) log.cc:844] Serialized 14 byte log entry
1026 14:59:26.358979 (+ 52683us) log.cc:844] Serialized 14 byte log entry
1026 14:59:26.549981 (+191002us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:26.550057 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:26.550102 (+ 45us) raft_consensus.cc:1685] finished
1026 14:59:26.550114 (+ 12us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:26.550240 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":115647,"prepare.queue_time_us":373343,"raft.queue_time_us":90985,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":9831,"thread_start_us":555224,"threads_started":3,"wal-append.queue_time_us":307783}
I20251026 14:59:26.552098 24332 consensus_queue.cc:1048] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20251026 14:59:26.549880 24031 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:26.555291 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:26.557039 24680 consensus_queue.cc:1048] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251026 14:59:26.557446 22799 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:26.539278 24837 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.558370 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:26.559120 22805 raft_consensus.cc:2368] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6.
I20251026 14:59:26.559492 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:26.560067 22799 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:26.554376 22804 rpcz_store.cc:276] 1026 14:59:25.530567 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:25.844052 (+313485us) service_pool.cc:225] Handling call
1026 14:59:25.844388 (+ 336us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:25.916180 (+ 71792us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:25.916218 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:25.934847 (+ 18629us) log.cc:844] Serialized 26 byte log entry
1026 14:59:26.205456 (+270609us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:26.512608 (+307152us) log.cc:844] Serialized 14 byte log entry
1026 14:59:26.530067 (+ 17459us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:26.530148 (+ 81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:26.550779 (+ 20631us) raft_consensus.cc:1685] finished
1026 14:59:26.550803 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:26.550944 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56636,"raft.queue_time_us":44508,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":95,"thread_start_us":402446,"threads_started":2,"wal-append.queue_time_us":533747}
I20251026 14:59:26.576124 24837 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:26.578186 24837 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:26.578915 22804 raft_consensus.cc:1275] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 14:59:26.578871 24837 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.581844 24837 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:26.582537 24837 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:26.583123 24837 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.585564 24837 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:26.588941 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:26.529964 24741 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:26.576014 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 200634us
W20251026 14:59:26.594169 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:26.595583 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:26.532097 24748 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.600432 22622 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:26.605101 22799 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
W20251026 14:59:26.492430 23399 rpcz_store.cc:269] 1026 14:59:24.516613 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:24.523528 (+ 6915us) service_pool.cc:225] Handling call
1026 14:59:26.362125 (+1838597us) spinlock_profiling.cc:243] Waited 242 ms on lock 0x7b5400470254. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:26.405407 (+ 43282us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":555682176}
I20251026 14:59:26.611577 22801 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:26.613152 22801 raft_consensus.cc:1275] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:26.607045 22799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1134) took 1024 ms. Trace:
I20251026 14:59:26.608798 24843 raft_consensus.cc:2804] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:26.609481 24873 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:26.609941 24840 consensus_queue.cc:1048] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 14:59:26.613986 22801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1135) took 1030 ms. Trace:
W20251026 14:59:26.627785 24836 consensus_meta.cc:287] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.619s user 0.010s sys 0.000s
I20251026 14:59:26.616479 24843 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:26.616640 24835 consensus_meta.cc:287] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 1.051s user 0.011s sys 0.000s
I20251026 14:59:26.617868 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:26.594645 24741 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.619652 24032 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:26.622150 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:26.623852 22622 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:26.627421 24895 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:26.614594 22799 rpcz_store.cc:276] 1026 14:59:25.583302 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:26.606292 (+1022990us) service_pool.cc:225] Handling call
1026 14:59:26.606585 (+ 293us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:26.606756 (+ 171us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:26.606779 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:26.606818 (+ 39us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:26.606843 (+ 25us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:26.606902 (+ 59us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:26.606991 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:26.634037 22801 rpcz_store.cc:276] 1026 14:59:25.583620 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:26.612407 (+1028787us) service_pool.cc:225] Handling call
1026 14:59:26.612768 (+ 361us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:26.613689 (+ 921us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:26.613917 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:26.648705 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:26.648958 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:26.636058 22804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1132) took 1053 ms. Trace:
I20251026 14:59:26.607800 24844 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:26.638382 24843 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:26.639815 22651 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:26.618641 24748 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:26.640256 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:26.646663 22656 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:26.628789 24781 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:26.505237 24876 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.638859 24835 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.668480 24891 consensus_queue.cc:1048] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:59:26.670012 24876 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:26.651350 22804 rpcz_store.cc:276] 1026 14:59:25.582523 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:26.579890 (+997367us) service_pool.cc:225] Handling call
1026 14:59:26.580177 (+ 287us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:26.580387 (+ 210us) raft_consensus.cc:1534] Early marking committed up to index 2
1026 14:59:26.584562 (+ 4175us) log.cc:844] Serialized 14 byte log entry
1026 14:59:26.635591 (+ 51029us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:26.635667 (+ 76us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:26.635726 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:26.635815 (+ 89us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:26.635985 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":47155,"threads_started":2}
I20251026 14:59:26.634552 24873 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.676996 22651 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:26.678786 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:26.681787 22773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.004s sys 0.017s
W20251026 14:59:26.682278 22773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.556s user 0.005s sys 0.017s
I20251026 14:59:26.684183 22773 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:26.686233 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:26.686676 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:26.690068 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:26.690886 24873 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:26.676642 24781 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.659991 24840 raft_consensus.cc:2804] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:26.696038 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:26.655830 24841 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:26.705792 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:26.671379 24748 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:26.671744 24898 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:26.645763 24836 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.670028 24835 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:26.657717 24895 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.711876 24840 raft_consensus.cc:697] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:26.713442 24895 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:26.714002 24895 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:26.714730 24741 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:26.715575 24741 consensus_queue.cc:1048] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:59:26.716919 24741 raft_consensus.cc:2804] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:26.717366 24741 raft_consensus.cc:697] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:26.714480 24895 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.723973 22656 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
W20251026 14:59:26.727277 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 76608.4us
I20251026 14:59:26.730993 22774 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:26.732805 22774 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:26.738245 22774 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:26.740651 22654 raft_consensus.cc:1240] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:26.719475 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:26.744966 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:26.746443 22656 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 7
I20251026 14:59:26.662410 24844 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.751374 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:26.752064 24906 raft_consensus.cc:2804] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:26.752534 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:26.752617 24906 raft_consensus.cc:697] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:26.753676 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:26.754029 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:26.712888 24840 consensus_queue.cc:237] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.758296 23400 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 LEARNER]: Advancing to term 6
I20251026 14:59:26.718199 24741 consensus_queue.cc:237] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.723650 24748 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.767999 23400 raft_consensus.cc:1275] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 LEARNER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20251026 14:59:26.787287 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 103218us
I20251026 14:59:26.800536 24844 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:26.801435 22803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1089) took 1277 ms. Trace:
I20251026 14:59:26.801860 22803 rpcz_store.cc:276] 1026 14:59:25.523816 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:25.686813 (+162997us) service_pool.cc:225] Handling call
1026 14:59:25.788447 (+101634us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:25.792029 (+ 3582us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:26.801207 (+1009178us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b5400389fd0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:26.801368 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":233012096}
I20251026 14:59:26.807127 22727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47996 (request call id 123) took 1269 ms. Trace:
I20251026 14:59:26.807570 22727 rpcz_store.cc:276] 1026 14:59:25.537320 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:25.537457 (+ 137us) service_pool.cc:225] Handling call
1026 14:59:25.537738 (+ 281us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:25.683660 (+145922us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:26.806892 (+1123232us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b540006b154. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:26.807064 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":299794176}
I20251026 14:59:26.808085 22651 raft_consensus.cc:3060] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:26.787467 24841 raft_consensus.cc:2804] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:26.809549 22727 raft_consensus.cc:1275] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251026 14:59:26.810649 22802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1102) took 1272 ms. Trace:
I20251026 14:59:26.811124 22802 rpcz_store.cc:276] 1026 14:59:25.538357 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:26.262747 (+724390us) service_pool.cc:225] Handling call
1026 14:59:26.268632 (+ 5885us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:26.301550 (+ 32918us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:26.301597 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:26.302172 (+ 575us) log.cc:844] Serialized 26 byte log entry
1026 14:59:26.595414 (+293242us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:26.719462 (+124048us) log.cc:844] Serialized 14 byte log entry
1026 14:59:26.719703 (+ 241us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:26.719804 (+ 101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:26.810456 (+ 90652us) raft_consensus.cc:1685] finished
1026 14:59:26.810475 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:26.810586 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32381,"prepare.queue_time_us":150217,"raft.queue_time_us":289138,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":5850,"spinlock_wait_cycles":9660416,"thread_start_us":312950,"threads_started":2,"wal-append.queue_time_us":431171}
I20251026 14:59:26.819265 22656 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:26.754156 24031 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.783263 24907 raft_consensus.cc:2749] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:26.852423 24836 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:26.809213 24843 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.854354 24906 consensus_queue.cc:237] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.723877 24898 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:26.876474 22622 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:26.881048 24032 raft_consensus.cc:2749] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:26.884413 24788 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:26.887499 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:26.890252 22656 raft_consensus.cc:1240] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20251026 14:59:26.889120 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:26.893034 24745 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:26.898921 22651 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:26.869916 24841 raft_consensus.cc:697] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
W20251026 14:59:26.900679 22728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 1305) took 1733 ms (client timeout 1542 ms). Trace:
W20251026 14:59:26.903445 22728 rpcz_store.cc:269] 1026 14:59:25.161411 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:25.161742 (+ 331us) service_pool.cc:225] Handling call
1026 14:59:26.894112 (+1732370us) spinlock_profiling.cc:243] Waited 455 ms on lock 0x7b540006ac54. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:26.894232 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1046404864}
I20251026 14:59:26.909471 24909 consensus_queue.cc:1048] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20251026 14:59:26.910285 24740 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:26.910804 24740 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.911780 24895 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:26.914314 24841 consensus_queue.cc:237] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.913100 24740 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:26.919139 24745 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:26.947955 24850 raft_consensus.cc:3055] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:26.930083 24781 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:26.930595 24895 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:26.955377 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 70454.4us
I20251026 14:59:26.956110 22774 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:26.958066 22799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1137) took 1281 ms. Trace:
W20251026 14:59:26.959189 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:26.918422 24862 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.946040 24748 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:26.951259 24850 raft_consensus.cc:740] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:26.958946 22799 rpcz_store.cc:276] 1026 14:59:25.677041 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:26.675612 (+998571us) service_pool.cc:225] Handling call
1026 14:59:26.675880 (+ 268us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:26.676057 (+ 177us) raft_consensus.cc:1534] Early marking committed up to index 3
1026 14:59:26.903725 (+227668us) log.cc:844] Serialized 14 byte log entry
1026 14:59:26.957631 (+ 53906us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:26.957688 (+ 57us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:26.957731 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:26.957820 (+ 89us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:26.957991 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3206,"prepare.queue_time_us":91521,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":3310,"thread_start_us":261147,"threads_started":2}
I20251026 14:59:26.975973 22799 raft_consensus.cc:1275] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251026 14:59:26.960709 24781 raft_consensus.cc:2804] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:26.978430 22659 raft_consensus.cc:1240] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20251026 14:59:26.980944 24898 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:26.981245 22802 raft_consensus.cc:1275] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251026 14:59:26.982059 23401 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40914 (request call id 71) took 1535 ms. Trace:
I20251026 14:59:26.953441 24745 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:26.990643 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 83788.8us
I20251026 14:59:26.960966 24895 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:26.982465 23401 rpcz_store.cc:276] 1026 14:59:25.446215 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:25.489295 (+ 43080us) service_pool.cc:225] Handling call
1026 14:59:26.981780 (+1492485us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b5400461754. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:26.981994 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":253796224}
I20251026 14:59:26.997815 24781 raft_consensus.cc:697] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
W20251026 14:59:27.012126 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:27.023437 22655 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1277) took 2277 ms (client timeout 2043 ms). Trace:
I20251026 14:59:27.004637 24862 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:27.017836 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:27.028887 24745 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:27.030413 24781 consensus_queue.cc:237] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:27.023948 22655 rpcz_store.cc:269] 1026 14:59:24.761351 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:24.761598 (+ 247us) service_pool.cc:225] Handling call
1026 14:59:27.016974 (+2255376us) spinlock_profiling.cc:243] Waited 296 ms on lock 0x7b54001d97d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:27.017134 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":680576384}
I20251026 14:59:27.039157 24895 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:27.044090 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:27.006968 24850 consensus_queue.cc:260] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.048333 24850 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:27.051262 24885 consensus_queue.cc:1048] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:59:27.054217 24862 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:27.054808 24862 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.055892 22657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1294) took 1554 ms. Trace:
I20251026 14:59:27.056264 22657 rpcz_store.cc:276] 1026 14:59:25.501651 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:25.501770 (+ 119us) service_pool.cc:225] Handling call
1026 14:59:25.544704 (+ 42934us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:26.539100 (+994396us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:27.055644 (+516544us) spinlock_profiling.cc:243] Waited 982 ms on lock 0x7b5400265f54. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:27.055827 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2258843392}
I20251026 14:59:27.056814 24862 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:27.058876 22805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1081) took 1547 ms (client timeout 1549 ms). Trace:
W20251026 14:59:27.059266 22805 rpcz_store.cc:269] 1026 14:59:25.511288 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:25.581065 (+ 69777us) service_pool.cc:225] Handling call
1026 14:59:27.058641 (+1477576us) spinlock_profiling.cc:243] Waited 918 ms on lock 0x7b5400333bd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:27.058792 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2110552448}
I20251026 14:59:27.060736 24895 raft_consensus.cc:1064] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821: attempting to promote NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 to VOTER
I20251026 14:59:27.061058 24844 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:27.061193 24895 raft_consensus.cc:1068] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821: Unable to promote non-voter 248493ec312e4398b828c6cb00e5c4b1: Illegal state: Leader has not yet committed an operation in its own term
I20251026 14:59:27.061632 24844 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.065677 22806 raft_consensus.cc:1240] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:27.067083 22806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1095) took 1537 ms. Trace:
I20251026 14:59:27.067457 22806 rpcz_store.cc:276] 1026 14:59:25.529932 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:25.688513 (+158581us) service_pool.cc:225] Handling call
1026 14:59:27.065385 (+1376872us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:27.066186 (+ 801us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:27.066849 (+ 663us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b5400333bd0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:27.067016 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2947835776}
I20251026 14:59:27.090473 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:27.096899 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:27.070873 24834 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.702s user 0.029s sys 0.030s
I20251026 14:59:27.103982 22774 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:27.107496 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:27.108487 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:27.109321 22651 raft_consensus.cc:2376] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8. Re-sending same reply.
W20251026 14:59:27.111039 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:27.113183 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:27.113015 24895 consensus_queue.cc:1059] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20251026 14:59:27.114980 24921 consensus_queue.cc:1048] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.014s
I20251026 14:59:27.115804 22802 raft_consensus.cc:2393] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 6: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:27.121380 22651 pending_rounds.cc:85] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29: Aborting all ops after (but not including) 2
I20251026 14:59:27.121843 22651 pending_rounds.cc:107] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20251026 14:59:27.122157 22651 raft_consensus.cc:2889] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251026 14:59:27.104317 24834 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.738s user 0.029s sys 0.030s
I20251026 14:59:27.127636 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:27.127844 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:27.129420 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:27.103359 24745 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:27.134380 22800 raft_consensus.cc:1240] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20251026 14:59:27.077934 24784 consensus_queue.cc:1048] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20251026 14:59:27.135720 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:27.117170 24885 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:27.126663 24759 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:27.125802 24906 raft_consensus.cc:2804] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:27.141803 22774 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:27.118175 24909 consensus_queue.cc:1059] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20251026 14:59:27.168351 24788 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:27.127439 24787 consensus_queue.cc:1059] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20251026 14:59:27.137682 24852 raft_consensus.cc:3055] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:27.201532 24852 raft_consensus.cc:740] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:27.139251 24745 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:27.208454 24932 log.cc:927] Time spent T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
W20251026 14:59:27.208690 24929 log.cc:927] Time spent T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20251026 14:59:27.205900 24852 consensus_queue.cc:260] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.210543 24852 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:27.211421 24745 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:27.212199 22802 raft_consensus.cc:1275] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:27.146127 24885 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.213855 24862 consensus_queue.cc:1048] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:59:27.215420 24885 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:27.217005 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:27.151527 24759 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.219834 24759 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:27.223070 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:27.223016 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:27.230263 24774 raft_consensus.cc:3055] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:27.236357 24774 raft_consensus.cc:740] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:27.237632 24788 raft_consensus.cc:697] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 8 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:27.237533 24774 consensus_queue.cc:260] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.239307 24774 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:27.239008 24788 consensus_queue.cc:237] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.243611 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:27.247927 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:27.254570 23403 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40914 (request call id 73) took 1284 ms (client timeout 1550 ms). Trace:
W20251026 14:59:27.255086 23403 rpcz_store.cc:269] 1026 14:59:25.970005 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:26.085946 (+115941us) service_pool.cc:225] Handling call
1026 14:59:27.222758 (+1136812us) spinlock_profiling.cc:243] Waited 2.58 ms on lock 0x7b58001e3c98. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8ce074451 00007fd8ce0747ce 000000000035a3c9 00007fd8d6091030 00007fd8d6159864 00007fd8d61598d1 00007fd8d6107660 00007fd8d6106f44 00007fd8cf6a4834 00007fd8cf6a47c2
1026 14:59:27.254504 (+ 31746us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5923072}
I20251026 14:59:27.256584 24787 raft_consensus.cc:3055] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:27.257033 24787 raft_consensus.cc:740] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:27.257900 24787 consensus_queue.cc:260] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.259065 24787 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:27.259308 22730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1311) took 1830 ms. Trace:
I20251026 14:59:27.259696 22730 rpcz_store.cc:276] 1026 14:59:25.428660 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:25.433472 (+ 4812us) service_pool.cc:225] Handling call
1026 14:59:25.440499 (+ 7027us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:26.063643 (+623144us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:27.259078 (+1195435us) spinlock_profiling.cc:243] Waited 566 ms on lock 0x7b540030acd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:27.259253 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1301484416}
W20251026 14:59:27.262516 22807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1085) took 1749 ms (client timeout 1681 ms). Trace:
W20251026 14:59:27.263077 22807 rpcz_store.cc:269] 1026 14:59:25.514258 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:25.614491 (+100233us) service_pool.cc:225] Handling call
1026 14:59:27.262251 (+1647760us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b540021bbd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:27.262432 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":616057088}
I20251026 14:59:27.279973 22807 raft_consensus.cc:1275] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:27.249926 24906 raft_consensus.cc:697] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:27.286450 24906 consensus_queue.cc:237] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.289861 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:27.287986 24031 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:27.305428 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 1.05734e+06us
I20251026 14:59:27.306457 22700 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:27.309130 24877 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:27.309763 24877 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.312005 24877 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:27.312526 24877 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:27.314720 24730 consensus_queue.cc:1048] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251026 14:59:27.318284 24849 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:27.318796 24850 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:27.319178 24766 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:27.318976 24849 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.319375 24850 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:59:27.317018 24737 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:27.320525 24737 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.321368 24849 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:27.321949 22801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1136) took 1667 ms. Trace:
I20251026 14:59:27.322417 22801 rpcz_store.cc:276] 1026 14:59:25.654312 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:26.650086 (+995774us) service_pool.cc:225] Handling call
1026 14:59:26.665627 (+ 15541us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:26.672259 (+ 6632us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:26.672290 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:26.672336 (+ 46us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:26.672366 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:27.321721 (+649355us) spinlock_profiling.cc:243] Waited 6.42 ms on lock 0x7b540020bcd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:27.321767 (+ 46us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:27.321891 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14765440}
I20251026 14:59:27.324182 22700 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:27.312997 24877 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.327085 24610 consensus_queue.cc:1048] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:59:27.328485 24610 raft_consensus.cc:2804] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:27.322459 24737 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:27.319707 24766 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.332780 24766 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:27.328928 24850 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:27.318221 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 60942.1us
I20251026 14:59:27.337344 24884 consensus_queue.cc:1048] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20251026 14:59:27.339172 24850 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:27.340487 24850 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.343114 24850 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:27.343779 24850 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:27.344305 24850 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.349701 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:27.350728 22728 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:27.352229 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:27.354432 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:27.350876 22804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1138) took 1664 ms. Trace:
I20251026 14:59:27.327309 24877 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:27.368036 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:27.369635 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:27.370824 22728 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:27.358582 22804 rpcz_store.cc:276] 1026 14:59:25.686178 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:26.671172 (+984994us) service_pool.cc:225] Handling call
1026 14:59:26.676660 (+ 5488us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:26.797789 (+121129us) raft_consensus.cc:1534] Early marking committed up to index 3
1026 14:59:26.849889 (+ 52100us) log.cc:844] Serialized 14 byte log entry
1026 14:59:26.905192 (+ 55303us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:26.906062 (+ 870us) log.cc:844] Serialized 26 byte log entry
1026 14:59:26.906186 (+ 124us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:26.906210 (+ 24us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:27.311324 (+405114us) spinlock_profiling.cc:243] Waited 2.72 ms on lock 0x7b18003c2e60. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce07f986 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9
1026 14:59:27.313177 (+ 1853us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b540020c1d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:27.313228 (+ 51us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:27.350630 (+ 37402us) raft_consensus.cc:1685] finished
1026 14:59:27.350655 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:27.350804 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30215,"prepare.queue_time_us":2157,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":48041,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":3402,"spinlock_wait_cycles":256427136,"thread_start_us":27454,"threads_started":1,"wal-append.queue_time_us":24891}
I20251026 14:59:27.375603 22728 raft_consensus.cc:1275] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20251026 14:59:27.396945 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 71739.8us
I20251026 14:59:27.321957 24849 raft_consensus.cc:2804] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:27.400985 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:27.401691 22730 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:27.328943 24610 raft_consensus.cc:697] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
W20251026 14:59:27.405870 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:27.407951 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:27.366688 24752 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:27.292114 24886 consensus_queue.cc:1048] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.117s
I20251026 14:59:27.500566 24742 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:27.503486 23402 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43606 (request call id 55) took 1735 ms. Trace:
I20251026 14:59:27.491307 24610 consensus_queue.cc:237] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:27.507825 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 165549us
I20251026 14:59:27.492182 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:27.492748 22657 raft_consensus.cc:1240] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:27.397351 24781 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:27.477411 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:27.535454 22727 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
W20251026 14:59:27.495127 24936 log.cc:927] Time spent T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.191s user 0.000s sys 0.000s
I20251026 14:59:27.407912 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:27.425889 22651 raft_consensus.cc:1240] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 6. Current term is 7. Ops: []
W20251026 14:59:27.526814 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:27.548686 22730 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:27.537863 22653 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:27.551430 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 967) took 1899 ms. Trace:
W20251026 14:59:27.551584 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 151214us
I20251026 14:59:27.566818 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:27.579411 22730 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:27.567236 22726 rpcz_store.cc:276] 1026 14:59:25.651729 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:25.844919 (+193190us) service_pool.cc:225] Handling call
1026 14:59:25.845188 (+ 269us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:25.845391 (+ 203us) raft_consensus.cc:1534] Early marking committed up to index 3
1026 14:59:25.919476 (+ 74085us) log.cc:844] Serialized 14 byte log entry
1026 14:59:27.551094 (+1631618us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:27.551152 (+ 58us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:27.551193 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:27.551262 (+ 69us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:27.551376 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1039786,"prepare.queue_time_us":73928,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":1050668,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":72,"thread_start_us":591374,"threads_started":2,"wal-append.queue_time_us":681350}
I20251026 14:59:27.515460 24844 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:27.480921 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:27.397938 24849 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:27.594786 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 961) took 1959 ms. Trace:
I20251026 14:59:27.581120 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:27.598266 22726 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 7
I20251026 14:59:27.550005 24806 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:27.593163 22728 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
W20251026 14:59:27.604791 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 1.47532e+06us
I20251026 14:59:27.615731 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:27.605556 22729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 959) took 1971 ms. Trace:
I20251026 14:59:27.615904 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:27.596004 22725 rpcz_store.cc:276] 1026 14:59:25.635787 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:25.640435 (+ 4648us) service_pool.cc:225] Handling call
1026 14:59:25.640750 (+ 315us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:25.650567 (+ 9817us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:25.650621 (+ 54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:25.658805 (+ 8184us) log.cc:844] Serialized 26 byte log entry
1026 14:59:26.113341 (+454536us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:26.134168 (+ 20827us) log.cc:844] Serialized 14 byte log entry
1026 14:59:26.712854 (+578686us) spinlock_profiling.cc:243] Waited 4.68 ms on lock 0x7b48001ec258. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d606b971 00007fd8d6062a37 00007fd8d60628ca 00007fd8d6159a0f 00007fd8d61596d7 00007fd8d616dcfc 00007fd8d616dca8 00007fd8d616dc28 00007fd8d616dbe3 00007fd8d616ce8c 00007fd8d50aa326
1026 14:59:27.539367 (+826513us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:27.539449 (+ 82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:27.539497 (+ 48us) raft_consensus.cc:1685] finished
1026 14:59:27.594442 (+ 54945us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:27.594698 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1772016,"prepare.queue_time_us":20458,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":61,"raft.queue_time_us":1191158,"raft.run_cpu_time_us":31330,"raft.run_wall_time_us":1059226,"spinlock_wait_cycles":10774400,"thread_start_us":563716,"threads_started":2,"wal-append.queue_time_us":555936,"wal-append.run_cpu_time_us":2139,"wal-append.run_wall_time_us":1004604}
I20251026 14:59:27.635082 22697 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:27.635250 22656 raft_consensus.cc:2410] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20251026 14:59:27.635972 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 123729us
I20251026 14:59:27.537760 24752 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.584388 24763 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:27.640389 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:27.604390 24766 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:27.583215 24877 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:27.641988 22725 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
W20251026 14:59:27.637111 24926 log.cc:927] Time spent T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.123s user 0.000s sys 0.000s
I20251026 14:59:27.647509 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:27.648181 22728 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:27.648958 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:27.627965 24936 log.cc:927] Time spent T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20251026 14:59:27.550494 24781 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.635707 22729 rpcz_store.cc:276] 1026 14:59:25.633788 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:25.633964 (+ 176us) service_pool.cc:225] Handling call
1026 14:59:25.634351 (+ 387us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:25.634581 (+ 230us) raft_consensus.cc:1534] Early marking committed up to index 4
1026 14:59:25.634849 (+ 268us) log.cc:844] Serialized 14 byte log entry
1026 14:59:26.308296 (+673447us) spinlock_profiling.cc:243] Waited 1.85 ms on lock 0x7b18003c4a80. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce0942f7 00007fd8d6e6d219 00007fd8ce0b0c4b 00007fd8ce0aeeef 00007fd8ce0b11b2 00007fd8d0cf82bb 00007fd8d0cfddc3 00007fd8d0cfe2da 00007fd8d11685ba
1026 14:59:27.605138 (+1296842us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:27.605198 (+ 60us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:27.605246 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:27.605340 (+ 94us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:27.605491 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":815934,"prepare.queue_time_us":98,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":3326,"raft.queue_time_us":824056,"raft.run_cpu_time_us":69,"raft.run_wall_time_us":69,"spinlock_wait_cycles":4244736,"thread_start_us":480553,"threads_started":2,"wal-append.queue_time_us":952245,"wal-append.run_cpu_time_us":996,"wal-append.run_wall_time_us":1001088}
I20251026 14:59:27.504045 23402 rpcz_store.cc:276] 1026 14:59:25.690460 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:25.880972 (+190512us) service_pool.cc:225] Handling call
1026 14:59:26.995802 (+1114830us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:26.996398 (+ 596us) raft_consensus.cc:1534] Early marking committed up to index 5
1026 14:59:26.996456 (+ 58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:26.997133 (+ 677us) log.cc:844] Serialized 26 byte log entry
1026 14:59:27.169407 (+172274us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:27.194432 (+ 25025us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:27.199276 (+ 4844us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:27.224425 (+ 25149us) raft_consensus.cc:1685] finished
1026 14:59:27.224445 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:27.345431 (+120986us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400461750. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:27.425371 (+ 79940us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1512,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":3050,"spinlock_wait_cycles":2493975808,"thread_start_us":147236,"threads_started":1,"wal-append.queue_time_us":199041}
I20251026 14:59:27.653265 22774 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:27.423583 24837 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:27.606824 24849 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:27.621500 22727 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:27.655756 22730 raft_consensus.cc:2468] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:27.657648 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:27.658511 22725 raft_consensus.cc:2468] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:27.661162 24851 consensus_queue.cc:1048] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.007s
I20251026 14:59:27.654830 24766 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:27.669545 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:27.673713 22726 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:27.676303 22697 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:27.679366 22697 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:27.685494 22659 raft_consensus.cc:1275] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20251026 14:59:27.689756 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:27.669872 24877 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:27.700091 22726 raft_consensus.cc:2468] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 2.
I20251026 14:59:27.635833 24806 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.712457 22725 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:27.650254 24763 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.698943 24872 raft_consensus.cc:2804] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:27.698789 24861 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:27.771096 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:27.772615 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:27.697096 24948 raft_consensus.cc:2749] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:27.776094 22728 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
W20251026 14:59:27.766476 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 45270.9us
I20251026 14:59:27.714424 24850 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:27.721748 24856 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:27.720175 24766 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:27.764530 24926 log.cc:927] Time spent T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
I20251026 14:59:27.682472 24837 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.759991 24872 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:27.827155 22725 raft_consensus.cc:1275] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:27.827049 24946 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:27.828649 24950 log.cc:927] Time spent T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20251026 14:59:27.828222 24946 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.829998 24877 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:27.830454 24946 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:27.848814 22733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1316) took 2064 ms. Trace:
I20251026 14:59:27.849433 22733 rpcz_store.cc:276] 1026 14:59:25.784452 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:26.098983 (+314531us) service_pool.cc:225] Handling call
1026 14:59:26.100215 (+ 1232us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:27.292807 (+1192592us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:27.292855 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:27.299431 (+ 6576us) log.cc:844] Serialized 26 byte log entry
1026 14:59:27.626981 (+327550us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:27.627115 (+ 134us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:27.627270 (+ 155us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:27.848498 (+221228us) raft_consensus.cc:1685] finished
1026 14:59:27.848531 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:27.848732 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1192078,"raft.queue_time_us":1204700,"spinlock_wait_cycles":69045120,"thread_start_us":198823,"threads_started":1,"wal-append.queue_time_us":379230}
W20251026 14:59:27.818941 24940 log.cc:927] Time spent T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.219s user 0.001s sys 0.001s
I20251026 14:59:27.786365 24921 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
W20251026 14:59:27.899142 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:27.902374 24852 consensus_meta.cc:287] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.691s user 0.011s sys 0.000s
I20251026 14:59:27.903658 22803 raft_consensus.cc:2410] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251026 14:59:27.817364 24850 raft_consensus.cc:2749] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:27.783715 24861 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.825255 24872 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:27.827297 24949 raft_consensus.cc:2749] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:27.816995 24835 consensus_queue.cc:1048] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.089s
W20251026 14:59:27.920879 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 241595us
I20251026 14:59:27.921703 22697 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:27.923864 22697 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:27.906739 24774 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:27.927246 22697 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:27.847889 24945 consensus_queue.cc:1048] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.137s
I20251026 14:59:27.935999 24837 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:27.917452 24921 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.940325 24921 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:27.941480 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:27.942508 22730 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:27.943733 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:27.944391 22799 raft_consensus.cc:2468] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:27.946337 22622 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:27.947912 24921 raft_consensus.cc:2804] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:27.948578 24921 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:27.949026 24921 raft_consensus.cc:3060] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:27.951426 22554 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50170 (request call id 60) took 8269 ms. Trace:
I20251026 14:59:27.050824 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:59:27.957171 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:27.961055 24877 raft_consensus.cc:2749] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:27.961997 24861 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:27.928897 24946 raft_consensus.cc:2804] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:27.962776 22697 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:27.965813 24774 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.968220 24774 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:27.815578 24856 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:27.976647 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:27.975666 22697 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:27.922084 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:27.983973 22733 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
W20251026 14:59:27.985633 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 165144us
I20251026 14:59:27.985972 22808 raft_consensus.cc:2410] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20251026 14:59:27.986073 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:27.990465 22697 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:27.992285 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:27.973644 24938 log.cc:927] Time spent T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.294s user 0.001s sys 0.000s
I20251026 14:59:27.972389 24850 raft_consensus.cc:2749] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:27.962800 24946 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:27.993722 22807 raft_consensus.cc:1275] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:27.947192 24843 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:27.980031 24737 raft_consensus.cc:2749] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:27.969465 24752 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:27.986411 24740 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:28.014421 22730 raft_consensus.cc:1275] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20251026 14:59:28.023180 22733 raft_consensus.cc:1275] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:27.951817 22554 rpcz_store.cc:276] 1026 14:59:19.681640 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:19.711388 (+ 29748us) service_pool.cc:225] Handling call
1026 14:59:27.951325 (+8239937us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 14:59:25.051016 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 14:59:25.154277 (+103261us) write_op.cc:432] Acquiring schema lock in shared mode
1026 14:59:25.156739 (+ 2462us) write_op.cc:435] Acquired schema lock
1026 14:59:25.156761 (+ 22us) tablet.cc:660] Decoding operations
1026 14:59:25.180980 (+ 24219us) write_op.cc:620] Acquiring the partition lock for write op
1026 14:59:25.181041 (+ 61us) write_op.cc:641] Partition lock acquired for write op
1026 14:59:25.181068 (+ 27us) tablet.cc:683] Acquiring locks for 4 operations
1026 14:59:25.181491 (+ 423us) tablet.cc:699] Row locks acquired
1026 14:59:25.181508 (+ 17us) write_op.cc:260] PREPARE: finished
1026 14:59:25.181662 (+ 154us) write_op.cc:270] Start()
1026 14:59:25.181788 (+ 126us) write_op.cc:276] Timestamp: P: 1761490765181615 usec, L: 0
1026 14:59:25.181807 (+ 19us) op_driver.cc:348] REPLICATION: starting
1026 14:59:25.349348 (+167541us) log.cc:844] Serialized 1709 byte log entry
1026 14:59:26.084647 (+735299us) spinlock_profiling.cc:243] Waited 41.2 ms on lock 0x7b440002304c. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8cdfc0905 00007fd8cdfc1322 00007fd8d0d0a231 00007fd8d0d05b5a 00007fd8ce09459c 00007fd8d6e6d219 00007fd8ce0b0c4b 00007fd8ce0aeeef 00007fd8ce0b11b2 00007fd8d0cf82bb
1026 14:59:26.757413 (+672766us) op_driver.cc:464] REPLICATION: finished
1026 14:59:26.800286 (+ 42873us) write_op.cc:301] APPLY: starting
1026 14:59:26.800379 (+ 93us) tablet.cc:1365] starting BulkCheckPresence
1026 14:59:26.800545 (+ 166us) tablet.cc:1368] finished BulkCheckPresence
1026 14:59:26.800571 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 14:59:26.801221 (+ 650us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 14:59:26.801443 (+ 222us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
1026 14:59:26.801499 (+ 56us) write_op.cc:312] APPLY: finished
1026 14:59:26.801741 (+ 242us) log.cc:844] Serialized 40 byte log entry
1026 14:59:26.801926 (+ 185us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:26.802038 (+ 112us) write_op.cc:454] Released schema lock
1026 14:59:26.802086 (+ 48us) write_op.cc:341] FINISH: Updating metrics
1026 14:59:26.802454 (+ 368us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:26.802481 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1220764,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":42585,"apply.run_cpu_time_us":2153,"apply.run_wall_time_us":2154,"num_ops":4,"prepare.queue_time_us":249129,"prepare.run_cpu_time_us":46035,"prepare.run_wall_time_us":1210481,"raft.queue_time_us":198517,"raft.run_cpu_time_us":6808,"raft.run_wall_time_us":73976,"replication_time_us":1569473,"spinlock_wait_cycles":94795520,"thread_start_us":1103963,"threads_started":4,"wal-append.queue_time_us":997958,"wal-append.run_cpu_time_us":24296,"wal-append.run_wall_time_us":1452394}]]}
I20251026 14:59:28.046948 24921 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:28.050791 22725 raft_consensus.cc:1275] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20251026 14:59:28.066430 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:28.067524 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:28.068836 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:28.053228 22654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1300) took 2545 ms. Trace:
I20251026 14:59:28.069660 22654 rpcz_store.cc:276] 1026 14:59:25.507993 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:25.605760 (+ 97767us) service_pool.cc:225] Handling call
1026 14:59:25.606250 (+ 490us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:26.741108 (+1134858us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:28.052944 (+1311836us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540012d9d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:28.053131 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2608296192}
I20251026 14:59:28.053978 22727 raft_consensus.cc:1275] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20251026 14:59:28.008090 24949 raft_consensus.cc:2749] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:28.058748 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:28.008044 24781 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:28.009927 24922 log.cc:927] Time spent T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1: Append to log took a long time: real 0.069s user 0.000s sys 0.001s
I20251026 14:59:28.039324 24952 raft_consensus.cc:2749] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:28.071897 22725 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:28.064098 22697 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:28.076601 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:28.084399 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:28.085831 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:28.017091 24946 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:28.086915 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 103208us
I20251026 14:59:28.087145 22697 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:28.084352 22727 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:28.088438 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:28.091218 22697 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:28.091218 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:28.085072 24937 log.cc:927] Time spent T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.457s user 0.001s sys 0.000s
I20251026 14:59:28.101734 22697 leader_election.cc:400] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6.
I20251026 14:59:28.107071 24948 consensus_queue.cc:1048] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:59:28.098493 24752 raft_consensus.cc:2749] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:28.099323 22725 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:28.103596 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:28.104782 22727 raft_consensus.cc:1275] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20251026 14:59:28.118353 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:28.118356 22697 leader_election.cc:400] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 6. Current term is 7.
I20251026 14:59:28.119365 22697 leader_election.cc:403] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20251026 14:59:28.084831 24740 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:28.121330 24945 consensus_queue.cc:1059] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:59:28.122228 24945 raft_consensus.cc:3055] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:28.085929 24830 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:28.124097 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:28.094624 24949 raft_consensus.cc:2749] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:28.102916 24763 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:28.122648 24945 raft_consensus.cc:740] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:28.098362 24954 raft_consensus.cc:2749] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:28.143543 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:28.139168 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:28.141036 22730 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:28.148404 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:28.149061 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 229518us
W20251026 14:59:28.151144 23369 leader_election.cc:336] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.546s (SENT)
I20251026 14:59:28.148423 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:28.152068 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:28.153123 22725 raft_consensus.cc:1275] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20251026 14:59:28.154538 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:28.155928 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:28.156623 22725 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
W20251026 14:59:28.140148 24956 log.cc:927] Time spent T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.079s user 0.000s sys 0.001s
I20251026 14:59:28.180675 22725 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
W20251026 14:59:28.161303 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:28.207538 22697 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:28.209484 22697 leader_election.cc:304] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:28.144413 24763 raft_consensus.cc:2749] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 6. Current term is 7.
W20251026 14:59:28.210058 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:28.210384 22800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1123) took 2783 ms. Trace:
I20251026 14:59:28.214093 22800 rpcz_store.cc:276] 1026 14:59:25.378817 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:25.379715 (+ 898us) service_pool.cc:225] Handling call
1026 14:59:25.417059 (+ 37344us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:27.135007 (+1717948us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:28.161715 (+1026708us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b540004c9d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:28.161947 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3948792192}
I20251026 14:59:28.216212 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:28.221694 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:28.223764 22697 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:28.224165 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:28.230456 22697 leader_election.cc:304] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:28.236738 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:28.162987 24948 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:28.247594 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:28.246191 24752 raft_consensus.cc:2749] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:28.263195 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:28.130965 24830 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:28.273579 22656 raft_consensus.cc:1275] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20251026 14:59:28.257083 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:28.276011 22654 raft_consensus.cc:1275] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251026 14:59:28.276657 22730 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:28.259268 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:28.202970 24856 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:28.250356 24936 log.cc:927] Time spent T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20251026 14:59:28.213187 24727 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:28.230348 24837 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:28.146325 24735 consensus_queue.cc:1048] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.088s
I20251026 14:59:28.213548 24954 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:28.232578 24949 raft_consensus.cc:2749] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:28.297963 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:28.300495 22799 raft_consensus.cc:1217] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 5.3->[6.4-6.4] Dedup: 6.4->[]
I20251026 14:59:28.254527 24964 raft_consensus.cc:2749] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:28.302549 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:28.314539 22651 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:28.303530 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:28.315797 24935 consensus_queue.cc:1048] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.106s
I20251026 14:59:28.317334 24806 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:28.318271 24970 log.cc:927] Time spent T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20251026 14:59:28.318828 24965 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:28.319331 24965 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:28.254860 24891 consensus_queue.cc:1059] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
I20251026 14:59:28.320815 24891 raft_consensus.cc:3055] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:28.321170 24891 raft_consensus.cc:740] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:28.321348 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:28.322082 24891 consensus_queue.cc:260] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:28.309027 22656 raft_consensus.cc:1275] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20251026 14:59:28.313807 22653 raft_consensus.cc:2393] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 5: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:28.316157 24857 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:28.345899 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:28.246191 24850 raft_consensus.cc:2749] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:28.346720 22657 raft_consensus.cc:2468] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:28.308439 22731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1317) took 2523 ms. Trace:
I20251026 14:59:28.325455 22799 raft_consensus.cc:1240] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20251026 14:59:28.292953 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:28.347502 22731 rpcz_store.cc:276] 1026 14:59:25.784759 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:26.404357 (+619598us) service_pool.cc:225] Handling call
1026 14:59:26.404734 (+ 377us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:27.307781 (+903047us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:27.307810 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:27.398757 (+ 90947us) log.cc:844] Serialized 26 byte log entry
1026 14:59:27.725026 (+326269us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:27.725084 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:27.725181 (+ 97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:28.308214 (+583033us) raft_consensus.cc:1685] finished
1026 14:59:28.308234 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:28.308370 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":902650,"raft.queue_time_us":911486,"raft.run_cpu_time_us":51,"raft.run_wall_time_us":52,"thread_start_us":181969,"threads_started":1,"wal-append.queue_time_us":547159}
I20251026 14:59:28.359589 24891 raft_consensus.cc:3060] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:28.359995 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:28.360625 22731 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 6.
I20251026 14:59:28.361131 24946 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:28.361865 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:28.364084 22657 raft_consensus.cc:2410] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251026 14:59:28.367514 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:28.268824 24948 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:28.346899 24921 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:28.304847 24727 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:28.228973 24945 consensus_queue.cc:260] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:28.369856 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:28.375308 22774 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:28.384001 22774 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:28.309075 24837 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:28.394204 22800 raft_consensus.cc:2468] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:28.395565 24837 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:28.396567 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:59:28.353103 24787 consensus_meta.cc:287] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 1.092s user 0.014s sys 0.009s
I20251026 14:59:28.385208 22731 raft_consensus.cc:2468] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 5.
I20251026 14:59:28.379482 24947 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:28.398043 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:28.397785 24947 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:28.399613 24963 log.cc:927] Time spent T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20251026 14:59:28.370105 24965 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:28.409966 24873 raft_consensus.cc:2804] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:28.410449 24873 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:28.410832 24873 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
W20251026 14:59:28.412385 24976 log.cc:927] Time spent T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.096s user 0.000s sys 0.001s
I20251026 14:59:28.370097 24625 consensus_queue.cc:1048] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.008s
W20251026 14:59:28.414408 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:28.414307 24625 consensus_queue.cc:1048] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:59:28.415747 24967 raft_consensus.cc:2804] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:28.416162 24967 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:28.416487 24967 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
W20251026 14:59:28.423430 24849 consensus_meta.cc:287] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.735s user 0.011s sys 0.000s
I20251026 14:59:28.423851 24849 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:28.426811 24830 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:28.428944 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:28.430195 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:28.432757 24948 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:28.433964 24945 raft_consensus.cc:3060] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:28.347432 24872 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:28.438787 22806 raft_consensus.cc:2393] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 9: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:28.439157 24872 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:28.355903 24857 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:28.441795 24857 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:28.323853 24954 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:28.389716 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1319) took 2536 ms. Trace:
I20251026 14:59:28.448648 22734 rpcz_store.cc:276] 1026 14:59:25.853483 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:26.547645 (+694162us) service_pool.cc:225] Handling call
1026 14:59:26.548062 (+ 417us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:27.760315 (+1212253us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:27.760380 (+ 65us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:27.766095 (+ 5715us) log.cc:844] Serialized 26 byte log entry
1026 14:59:28.019006 (+252911us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:28.019067 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:28.019167 (+ 100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:28.389389 (+370222us) raft_consensus.cc:1685] finished
1026 14:59:28.389426 (+ 37us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:28.389640 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":760497,"raft.queue_time_us":768374,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"thread_start_us":593298,"threads_started":2,"wal-append.queue_time_us":473347}
W20251026 14:59:28.387473 24969 log.cc:927] Time spent T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
I20251026 14:59:28.397167 22800 raft_consensus.cc:3060] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:28.444118 22731 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
W20251026 14:59:28.444168 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 94513.9us
W20251026 14:59:28.457052 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:28.462182 24849 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Requested vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:59:28.370560 24906 consensus_queue.cc:1048] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.084s
I20251026 14:59:28.484599 22651 raft_consensus.cc:1240] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:28.485299 22658 raft_consensus.cc:1275] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:28.454702 24532 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:28.492074 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:28.499258 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:28.501494 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:28.502267 22652 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:28.537124 22733 raft_consensus.cc:1275] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20251026 14:59:28.537613 22733 pending_rounds.cc:85] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e: Aborting all ops after (but not including) 3
I20251026 14:59:28.537870 22733 pending_rounds.cc:107] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20251026 14:59:28.538134 22733 raft_consensus.cc:2889] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:28.539774 22733 raft_consensus.cc:1275] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:28.508137 24872 consensus_queue.cc:1048] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
W20251026 14:59:28.556160 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 113736us
I20251026 14:59:28.557112 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:28.508080 24787 consensus_queue.cc:1059] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20251026 14:59:28.504498 24532 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:28.556219 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 115966us
I20251026 14:59:28.561561 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:28.577540 22697 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
W20251026 14:59:28.579741 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:28.580574 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:28.583987 22805 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:28.585314 22622 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:28.589134 22697 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:28.589015 22652 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 9.
I20251026 14:59:28.592314 22774 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:28.595574 24948 raft_consensus.cc:2804] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:28.591529 24527 raft_consensus.cc:2804] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:28.591218 24854 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:28.612140 22773 leader_election.cc:336] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.542s (SENT)
I20251026 14:59:28.634264 22773 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:28.598206 24973 log.cc:927] Time spent T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.140s user 0.003s sys 0.001s
I20251026 14:59:28.591003 24806 raft_consensus.cc:2749] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251026 14:59:28.615000 22697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:28.638885 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:28.638877 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:28.639751 22805 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:28.600626 22800 raft_consensus.cc:2468] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:28.602061 24593 raft_consensus.cc:2804] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader election won for term 9
I20251026 14:59:28.613461 24948 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:28.641665 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:28.645043 22622 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:28.610842 24946 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:28.615322 24527 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:28.610357 24847 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:28.645826 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:28.651701 24978 log.cc:927] Time spent T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.200s user 0.001s sys 0.000s
I20251026 14:59:28.663990 24727 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:28.668156 24988 log.cc:927] Time spent T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.071s user 0.001s sys 0.000s
I20251026 14:59:28.657397 24948 raft_consensus.cc:3060] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:28.698894 22805 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:28.669448 24527 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:28.699211 23369 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:28.645020 24834 raft_consensus.cc:2749] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:28.680406 24853 raft_consensus.cc:2804] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:28.709898 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:28.722400 22697 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:28.665918 24593 raft_consensus.cc:697] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 9 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:28.650637 24854 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:28.732362 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:28.705159 22773 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:28.736485 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:28.737948 22773 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:28.686771 24946 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:28.739892 24855 raft_consensus.cc:2804] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20251026 14:59:28.740437 24985 log.cc:927] Time spent T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.088s user 0.000s sys 0.001s
I20251026 14:59:28.678642 24847 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:28.763523 24855 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:28.764017 24947 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:28.764037 24855 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:28.765239 24847 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:28.740806 24834 raft_consensus.cc:2749] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:28.744100 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:28.736388 24849 raft_consensus.cc:2804] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:28.767309 24849 raft_consensus.cc:697] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:28.767458 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:28.748044 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:28.769832 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:28.750645 24984 log.cc:927] Time spent T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.136s user 0.000s sys 0.000s
W20251026 14:59:28.751832 24989 log.cc:927] Time spent T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.141s user 0.001s sys 0.000s
I20251026 14:59:28.753623 24906 raft_consensus.cc:1064] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821: attempting to promote NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 to VOTER
W20251026 14:59:28.754880 24977 log.cc:927] Time spent T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.433s user 0.000s sys 0.004s
I20251026 14:59:28.754954 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:28.756438 24854 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:28.789531 24987 log.cc:927] Time spent T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.145s user 0.001s sys 0.000s
I20251026 14:59:28.789804 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:28.790768 24740 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:28.757539 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:28.791523 24781 raft_consensus.cc:3055] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:28.791923 24781 raft_consensus.cc:740] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:28.738298 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:28.796927 22808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1092) took 3272 ms (client timeout 3032 ms). Trace:
I20251026 14:59:28.770881 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:28.771586 22624 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:28.774343 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:28.797909 22806 raft_consensus.cc:2393] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 7: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:28.800524 22624 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:28.802574 22804 raft_consensus.cc:1275] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 3. (term mismatch)
I20251026 14:59:28.802965 22804 pending_rounds.cc:85] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821: Aborting all ops after (but not including) 2
I20251026 14:59:28.803263 22804 pending_rounds.cc:107] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20251026 14:59:28.803499 22804 raft_consensus.cc:2889] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:28.800344 24967 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:28.805858 22800 raft_consensus.cc:2368] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6.
I20251026 14:59:28.807932 22806 raft_consensus.cc:1275] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251026 14:59:28.776217 22700 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:28.730645 24955 raft_consensus.cc:2749] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:28.778417 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:28.779016 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:28.779590 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:28.786348 24849 consensus_queue.cc:237] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:28.787818 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:28.768929 24946 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:28.797387 22808 rpcz_store.cc:269] 1026 14:59:25.524602 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:25.679898 (+155296us) service_pool.cc:225] Handling call
1026 14:59:28.796676 (+3116778us) spinlock_profiling.cc:243] Waited 2.3 s on lock 0x7b5400389ad4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:28.796814 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5300889856}
I20251026 14:59:28.806844 24851 raft_consensus.cc:2749] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:28.817447 24593 consensus_queue.cc:237] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:28.818523 24532 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:28.781867 24853 raft_consensus.cc:697] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:28.826828 22624 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:28.813078 24947 raft_consensus.cc:2749] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:28.828106 22656 raft_consensus.cc:1240] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 8. Current term is 9. Ops: []
I20251026 14:59:28.822489 22700 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:28.833047 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:28.834758 22655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 1113) took 1310 ms. Trace:
I20251026 14:59:28.835156 22655 rpcz_store.cc:276] 1026 14:59:27.524927 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:28.807135 (+1282208us) service_pool.cc:225] Handling call
1026 14:59:28.834681 (+ 27546us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:28.840442 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 102608us
I20251026 14:59:28.841946 22773 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:28.842430 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:28.820991 24854 raft_consensus.cc:2749] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:28.792781 24781 consensus_queue.cc:260] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:28.861845 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:28.864166 22733 raft_consensus.cc:2376] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7. Re-sending same reply.
I20251026 14:59:28.844956 24730 raft_consensus.cc:2749] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:28.858418 22773 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:28.868402 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:28.898546 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 59639.7us
I20251026 14:59:28.825937 24877 consensus_queue.cc:1059] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20251026 14:59:28.849149 24967 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:28.905613 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 53007.5us
I20251026 14:59:28.910108 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:28.857523 24834 raft_consensus.cc:2804] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:28.911104 22733 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:28.863773 24882 consensus_queue.cc:1048] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20251026 14:59:28.918360 24877 consensus_queue.cc:1243] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer df3a0d1619ee48e8ab68ee0543a72a29 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20251026 14:59:28.923606 22624 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:28.935882 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 1340) took 1251 ms. Trace:
I20251026 14:59:28.936458 22726 rpcz_store.cc:276] 1026 14:59:27.684821 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:27.689472 (+ 4651us) service_pool.cc:225] Handling call
1026 14:59:28.935647 (+1246175us) spinlock_profiling.cc:243] Waited 9.66 ms on lock 0x7b5400076f54. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:28.935813 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22219520}
I20251026 14:59:28.937912 22624 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:28.937947 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:28.940369 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:28.941215 22726 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:28.850385 24949 consensus_queue.cc:1048] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.065s
I20251026 14:59:28.942279 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:28.914989 24997 log.cc:927] Time spent T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.084s user 0.000s sys 0.001s
I20251026 14:59:28.944352 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:28.966145 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:28.966910 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:28.967707 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:28.944453 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:28.950492 24993 log.cc:927] Time spent T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.171s user 0.000s sys 0.000s
I20251026 14:59:28.950574 24851 raft_consensus.cc:2749] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:28.953532 24853 consensus_queue.cc:237] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:28.974478 24882 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:28.956189 24946 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:28.975198 24882 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:28.958468 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:28.861541 24031 consensus_queue.cc:1048] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20251026 14:59:28.912556 24781 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:28.961374 22726 raft_consensus.cc:2468] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:28.969022 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:28.977980 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:28.939525 24850 consensus_queue.cc:1059] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 5, Time since last communication: 0.028s
I20251026 14:59:28.987088 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 968) took 3333 ms. Trace:
I20251026 14:59:28.988390 24948 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:28.994194 24891 consensus_meta.cc:287] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.634s user 0.009s sys 0.003s
I20251026 14:59:28.906551 24857 consensus_queue.cc:1048] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
I20251026 14:59:29.000239 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:29.001493 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:29.003433 24831 raft_consensus.cc:2804] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election won for term 6
W20251026 14:59:29.004724 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 66929.9us
I20251026 14:59:29.008740 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:29.067523 22653 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:29.010138 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:28.919848 24834 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:29.011755 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:29.022641 22697 leader_election.cc:400] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6.
I20251026 14:59:29.069675 22697 leader_election.cc:403] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20251026 14:59:29.023334 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:29.069720 22655 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:29.027866 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:29.030145 22700 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:29.028501 24967 raft_consensus.cc:2749] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:29.031194 24840 raft_consensus.cc:2749] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:29.076720 22624 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:29.051386 22732 rpcz_store.cc:276] 1026 14:59:25.653205 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:25.854697 (+201492us) service_pool.cc:225] Handling call
1026 14:59:25.897758 (+ 43061us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:25.898337 (+ 579us) raft_consensus.cc:1534] Early marking committed up to index 2
1026 14:59:26.173785 (+275448us) log.cc:844] Serialized 14 byte log entry
1026 14:59:27.285161 (+1111376us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b180035bd20. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce026caa 00007fd8d7698a29 00007fd8d502965f 00007fd8ce0b1b0e 00007fd8ce0b3d61 00007fd8ce0b3d19 00007fd8ce0b3ca9 00007fd8ce0b3c71 00007fd8ce0b2f6d
1026 14:59:27.568444 (+283283us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:28.098189 (+529745us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b2c0000e678. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cfa5d9b1 00007fd8cfa6cfe2 00007fd8d1199701 00007fd8d115dfda 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22
1026 14:59:28.110377 (+ 12188us) log.cc:844] Serialized 52 byte log entry
1026 14:59:28.268628 (+158251us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:28.309038 (+ 40410us) log.cc:844] Serialized 14 byte log entry
1026 14:59:28.309406 (+ 368us) log.cc:844] Serialized 14 byte log entry
1026 14:59:28.313168 (+ 3762us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:28.313245 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:28.984293 (+671048us) spinlock_profiling.cc:243] Waited 3.34 ms on lock 0x7b180035bd20. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce0b1992 00007fd8ce0b3d61 00007fd8ce0b3d19 00007fd8ce0b3ca9 00007fd8ce0b3c71 00007fd8ce0b2f6d 0000000000379254 0000000000379099 00007fd8ce094d1a
1026 14:59:28.986845 (+ 2552us) raft_consensus.cc:1685] finished
1026 14:59:28.986867 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:28.987004 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":883095,"prepare.queue_time_us":352627,"prepare.run_cpu_time_us":146,"prepare.run_wall_time_us":135,"raft.queue_time_us":876206,"raft.run_cpu_time_us":742,"raft.run_wall_time_us":145996,"spinlock_wait_cycles":65873664,"thread_start_us":917914,"threads_started":5,"wal-append.queue_time_us":563858,"wal-append.run_cpu_time_us":1314,"wal-append.run_wall_time_us":1008408}
W20251026 14:59:29.081365 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 58856.2us
I20251026 14:59:29.018551 24527 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:29.039127 24946 consensus_queue.cc:1048] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20251026 14:59:29.090879 22731 raft_consensus.cc:1240] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:29.079169 24834 raft_consensus.cc:3060] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:29.105265 22651 raft_consensus.cc:1240] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: []
W20251026 14:59:29.096386 24873 consensus_meta.cc:287] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.685s user 0.011s sys 0.001s
W20251026 14:59:29.113983 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:29.117673 24873 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:29.119835 24873 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:29.121625 22805 raft_consensus.cc:1240] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 6. Current term is 7. Ops: []
W20251026 14:59:29.121783 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 91760.3us
I20251026 14:59:29.119552 24855 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:29.099236 24877 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:29.122488 22700 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:29.130491 22653 raft_consensus.cc:2468] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:29.135386 22700 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:29.139787 22801 raft_consensus.cc:1240] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:29.125304 24861 raft_consensus.cc:2749] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251026 14:59:29.128062 24949 raft_consensus.cc:2749] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:29.153363 24906 consensus_queue.cc:237] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 4, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:29.159907 25000 log.cc:927] Time spent T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
W20251026 14:59:29.162204 22684 heartbeater.cc:646] Failed to heartbeat to 127.21.2.62:38611 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.21.2.62:38611 timed out after 15.000s (SENT)
I20251026 14:59:29.138703 24727 raft_consensus.cc:2749] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251026 14:59:29.179227 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 43947.9us
I20251026 14:59:29.149389 24806 raft_consensus.cc:2749] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:29.186937 22700 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:29.202108 22655 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:29.209367 22700 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:29.210299 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 351934us
I20251026 14:59:29.211692 22700 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:29.224222 22700 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:29.212778 22773 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:29.241070 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:29.194259 24949 raft_consensus.cc:2749] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:29.242004 22654 raft_consensus.cc:1240] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:29.243389 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:59:29.250262 22773 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:29.245976 25000 log.cc:927] Time spent T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20251026 14:59:29.238133 24727 raft_consensus.cc:2749] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:29.242063 24806 raft_consensus.cc:2749] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:29.269181 22773 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:29.269747 22700 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:29.247056 24882 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:29.270947 22807 raft_consensus.cc:1240] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 9 LEADER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 8. Current term is 9. Ops: []
I20251026 14:59:29.258111 24967 raft_consensus.cc:2804] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:29.299060 22700 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:29.299638 22773 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:29.260303 24948 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:29.269011 24849 raft_consensus.cc:2749] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:29.269459 24527 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:29.303735 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 244361us
I20251026 14:59:29.298987 24967 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:29.309253 22773 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:29.310572 22773 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:29.316717 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 240093us
I20251026 14:59:29.318578 22773 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:29.300014 24593 raft_consensus.cc:2804] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:29.334033 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:29.271215 24831 raft_consensus.cc:697] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:29.281499 24949 consensus_queue.cc:1059] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.033s
I20251026 14:59:29.299918 25005 raft_consensus.cc:2804] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:29.352442 22624 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:29.300096 24806 raft_consensus.cc:2749] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:29.316378 24967 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:29.355285 22624 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:29.319319 24873 raft_consensus.cc:2804] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:29.320071 24855 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:29.413225 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 103945us
I20251026 14:59:29.415613 22700 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:29.342319 24593 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:29.353190 24965 raft_consensus.cc:2804] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:29.343066 24849 raft_consensus.cc:2749] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:29.352195 25005 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:29.335034 25006 raft_consensus.cc:2804] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:29.334959 24872 raft_consensus.cc:2749] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:29.342094 24882 consensus_queue.cc:1048] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.043s
W20251026 14:59:29.387094 25007 log.cc:927] Time spent T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
W20251026 14:59:29.398768 25004 log.cc:927] Time spent T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.250s user 0.000s sys 0.001s
W20251026 14:59:29.414270 25009 log.cc:927] Time spent T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.105s user 0.000s sys 0.001s
I20251026 14:59:29.414937 24873 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:29.537961 24873 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:29.540664 24877 raft_consensus.cc:2749] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6.
I20251026 14:59:29.414969 25013 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:29.421037 24806 raft_consensus.cc:3055] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 8 LEADER]: Stepping down as leader of term 8
I20251026 14:59:29.424409 24965 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:29.531178 24880 raft_consensus.cc:2804] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:29.461423 25006 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:29.564976 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 246449us
W20251026 14:59:29.569725 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:29.571805 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 49801.6us
I20251026 14:59:29.572939 22700 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:29.520977 25005 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:29.536772 24849 raft_consensus.cc:2749] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:29.582300 22773 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:29.421957 24593 raft_consensus.cc:3060] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:29.589988 22773 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:29.579749 25012 log.cc:927] Time spent T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.167s user 0.000s sys 0.001s
I20251026 14:59:29.563199 25013 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:59:29.599870 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 278379us
I20251026 14:59:29.570781 24965 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:29.606799 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:29.565399 24806 raft_consensus.cc:740] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:29.578817 24880 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:29.614753 22700 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:29.586408 24872 consensus_queue.cc:1048] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20251026 14:59:29.578289 25006 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:29.580539 25018 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:29.618880 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 263733us
I20251026 14:59:29.586162 24877 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:29.620369 22727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 999) took 1418 ms. Trace:
I20251026 14:59:29.629177 22727 rpcz_store.cc:276] 1026 14:59:28.202005 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:28.777463 (+575458us) service_pool.cc:225] Handling call
1026 14:59:28.780814 (+ 3351us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:28.781205 (+ 391us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:28.781233 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:28.803249 (+ 22016us) log.cc:844] Serialized 26 byte log entry
1026 14:59:28.903086 (+ 99837us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:28.903141 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:28.903218 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:29.620049 (+716831us) raft_consensus.cc:1685] finished
1026 14:59:29.620081 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:29.620286 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":48,"raft.queue_time_us":562,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":17821,"thread_start_us":64530,"threads_started":1,"wal-append.queue_time_us":284838}
I20251026 14:59:29.591691 24835 raft_consensus.cc:2749] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251026 14:59:29.631618 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 48095.9us
I20251026 14:59:29.633385 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47996 (request call id 129) took 1312 ms. Trace:
I20251026 14:59:29.631845 22624 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:29.610715 25013 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:29.633774 22725 rpcz_store.cc:276] 1026 14:59:28.320729 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:28.790834 (+470105us) service_pool.cc:225] Handling call
1026 14:59:28.791155 (+ 321us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:28.791553 (+ 398us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:28.791587 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:28.792176 (+ 589us) log.cc:844] Serialized 52 byte log entry
1026 14:59:29.205749 (+413573us) raft_consensus.cc:1645] Marking committed up to 7
1026 14:59:29.413457 (+207708us) log.cc:844] Serialized 14 byte log entry
1026 14:59:29.413814 (+ 357us) log.cc:844] Serialized 14 byte log entry
1026 14:59:29.421434 (+ 7620us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:29.421515 (+ 81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:29.633132 (+211617us) raft_consensus.cc:1685] finished
1026 14:59:29.633153 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:29.633328 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3599,"prepare.queue_time_us":601083,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":55,"raft.queue_time_us":125076,"raft.run_cpu_time_us":440,"raft.run_wall_time_us":55987,"thread_start_us":582096,"threads_started":2,"wal-append.queue_time_us":456348}
I20251026 14:59:29.616648 24880 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:29.599676 24895 consensus_queue.cc:1048] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.030s
I20251026 14:59:29.614583 25026 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:29.620507 24882 raft_consensus.cc:2749] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:29.666337 24835 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:29.666981 24835 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:29.670111 22774 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:29.685336 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 95457.4us
I20251026 14:59:29.691217 25014 raft_consensus.cc:2749] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:29.633061 25027 raft_consensus.cc:2749] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251026 14:59:29.747009 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 50334.3us
I20251026 14:59:29.750551 22773 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:29.635190 24947 raft_consensus.cc:2749] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:29.634232 24877 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:29.686362 24855 raft_consensus.cc:2804] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:29.423116 24831 consensus_queue.cc:237] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:29.633193 25018 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:29.752391 25007 log.cc:927] Time spent T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
W20251026 14:59:29.774691 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 142929us
I20251026 14:59:29.776770 24855 raft_consensus.cc:697] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:29.669102 25026 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:29.669601 24806 consensus_queue.cc:260] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:29.781208 22656 raft_consensus.cc:1275] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:29.780601 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:29.780767 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:29.782487 22659 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:29.778362 22624 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:29.769625 25030 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:29.772548 25027 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:29.794206 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:29.795202 22657 raft_consensus.cc:2468] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:29.777568 24949 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:29.781852 24835 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:29.800979 24031 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:29.780009 24872 raft_consensus.cc:2804] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:29.803951 22700 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:29.804461 24031 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:29.806550 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 56094us
I20251026 14:59:29.813242 22773 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:29.810971 24872 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:29.814435 24872 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
W20251026 14:59:29.816005 24962 log.cc:927] Time spent T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 1.443s user 0.002s sys 0.014s
I20251026 14:59:29.818079 22729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1334) took 2415 ms. Trace:
I20251026 14:59:29.818709 25036 consensus_queue.cc:1048] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20251026 14:59:29.797192 25027 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:29.822253 25026 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:29.818570 22729 rpcz_store.cc:276] 1026 14:59:27.402154 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:27.672080 (+269926us) service_pool.cc:225] Handling call
1026 14:59:27.672492 (+ 412us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:27.672886 (+ 394us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:27.672923 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:27.673491 (+ 568us) log.cc:844] Serialized 26 byte log entry
1026 14:59:27.985246 (+311755us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:27.985311 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:27.985379 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:29.578567 (+1593188us) spinlock_profiling.cc:243] Waited 4.72 ms on lock 0x7b440002304c. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8cdfc0905 00007fd8cdfc1322 00007fd8d0d0a231 00007fd8d0d05b5a 00007fd8d0cf991e 00007fd8d0cf8c66 00007fd8d0cf8888 00007fd8d0d022c1 00007fd8d0d02279 00007fd8d0d02209
1026 14:59:29.817705 (+239138us) raft_consensus.cc:1685] finished
1026 14:59:29.817738 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:29.817988 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4959,"raft.queue_time_us":30905,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":49487,"spinlock_wait_cycles":10860416,"thread_start_us":276744,"threads_started":1,"wal-append.queue_time_us":471018}
I20251026 14:59:29.820335 25036 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:29.823164 24855 consensus_queue.cc:237] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:29.823807 25036 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:29.829490 25031 raft_consensus.cc:2749] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251026 14:59:29.829741 22731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1003) took 1465 ms. Trace:
W20251026 14:59:29.812474 24834 consensus_meta.cc:287] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.691s user 0.010s sys 0.002s
I20251026 14:59:29.830323 22731 rpcz_store.cc:276] 1026 14:59:28.364129 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:28.819894 (+455765us) service_pool.cc:225] Handling call
1026 14:59:28.820197 (+ 303us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:29.097168 (+276971us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:29.829380 (+732212us) spinlock_profiling.cc:243] Waited 270 ms on lock 0x7b54000b95d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:29.829638 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":622026496}
I20251026 14:59:29.831228 24834 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:29.832636 22808 raft_consensus.cc:1240] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:29.809301 22659 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:29.846453 24031 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
W20251026 14:59:29.835536 25019 log.cc:927] Time spent T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.236s user 0.001s sys 0.000s
I20251026 14:59:29.856073 22773 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:29.822293 25027 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:29.842821 24532 raft_consensus.cc:2749] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251026 14:59:29.872661 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 94418.4us
I20251026 14:59:29.874300 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:29.875356 22773 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:29.876044 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:29.877657 22729 raft_consensus.cc:2376] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6. Re-sending same reply.
I20251026 14:59:29.818593 25035 consensus_queue.cc:1048] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.095s
I20251026 14:59:29.882249 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:29.880316 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:29.883183 22729 raft_consensus.cc:2376] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6. Re-sending same reply.
I20251026 14:59:29.810555 24949 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:29.801107 25030 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:29.885861 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:29.886463 22725 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:29.887248 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:29.864928 25048 raft_consensus.cc:2804] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:29.890343 22729 raft_consensus.cc:3055] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:29.873529 25047 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:29.895442 22725 raft_consensus.cc:2410] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251026 14:59:29.895318 22729 raft_consensus.cc:740] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:29.892467 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:29.879179 24891 raft_consensus.cc:2749] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:29.897648 22729 consensus_queue.cc:260] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:29.901736 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:30.012843 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:30.022573 22654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1126) took 1120 ms. Trace:
I20251026 14:59:29.884282 24834 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:29.996894 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:30.053884 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:30.058022 25010 log.cc:927] Time spent T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.659s user 0.000s sys 0.003s
I20251026 14:59:30.051469 24849 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:29.929981 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:30.063127 22732 raft_consensus.cc:2468] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:29.909866 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:29.942059 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1010) took 1348 ms. Trace:
W20251026 14:59:29.913390 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 155227us
I20251026 14:59:30.054847 22805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1134) took 1361 ms. Trace:
I20251026 14:59:30.055712 24949 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:29.943706 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:30.025009 22729 raft_consensus.cc:3060] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:29.903977 22731 raft_consensus.cc:3060] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:30.065992 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:30.064306 22805 rpcz_store.cc:276] 1026 14:59:28.693141 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:28.824465 (+131324us) service_pool.cc:225] Handling call
1026 14:59:28.824685 (+ 220us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:29.122042 (+297357us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:30.054572 (+932530us) spinlock_profiling.cc:243] Waited 297 ms on lock 0x7b54000c4754. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:30.054765 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":682425088}
I20251026 14:59:30.051033 22654 rpcz_store.cc:276] 1026 14:59:28.902330 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:28.902484 (+ 154us) service_pool.cc:225] Handling call
1026 14:59:28.902794 (+ 310us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:29.248576 (+345782us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:30.022255 (+773679us) spinlock_profiling.cc:243] Waited 339 ms on lock 0x7b54001fb154. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:30.022492 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":779603200}
I20251026 14:59:30.067396 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:30.067967 22725 raft_consensus.cc:2468] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:30.068732 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:29.891197 24954 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:30.074841 25008 log.cc:927] Time spent T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.691s user 0.002s sys 0.000s
I20251026 14:59:30.078184 22728 raft_consensus.cc:3055] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Stepping down as leader of term 6
W20251026 14:59:30.078600 24873 consensus_meta.cc:287] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.540s user 0.010s sys 0.000s
I20251026 14:59:30.079030 24873 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.081459 22804 raft_consensus.cc:1240] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:30.083083 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:30.084313 24880 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.078610 22728 raft_consensus.cc:740] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
W20251026 14:59:30.071153 25028 log.cc:927] Time spent T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.471s user 0.001s sys 0.000s
I20251026 14:59:30.071327 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:30.072540 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1132) took 1072 ms. Trace:
I20251026 14:59:30.070221 22805 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714"
dest_uuid: "0d684445287b450a8f738b6ad4b59821"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44570
I20251026 14:59:30.062076 24849 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:30.019968 24967 consensus_meta.cc:287] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.621s user 0.011s sys 0.000s
I20251026 14:59:30.007014 25026 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:30.088855 22651 rpcz_store.cc:276] 1026 14:59:29.000135 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:29.000769 (+ 634us) service_pool.cc:225] Handling call
1026 14:59:29.001050 (+ 281us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:29.105842 (+104792us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:30.072293 (+966451us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b54001d8dd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:30.072473 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":239212288}
I20251026 14:59:29.893515 25048 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:30.089402 22805 raft_consensus.cc:606] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Received request to transfer leadership
I20251026 14:59:30.063925 22726 rpcz_store.cc:276] 1026 14:59:28.593742 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:29.041366 (+447624us) service_pool.cc:225] Handling call
1026 14:59:29.041704 (+ 338us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:29.042110 (+ 406us) raft_consensus.cc:1534] Early marking committed up to index 3
1026 14:59:29.074268 (+ 32158us) log.cc:844] Serialized 14 byte log entry
1026 14:59:29.311733 (+237465us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:29.312197 (+ 464us) log.cc:844] Serialized 52 byte log entry
1026 14:59:29.312290 (+ 93us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:29.312440 (+ 150us) log.cc:844] Serialized 14 byte log entry
1026 14:59:29.663791 (+351351us) log.cc:844] Serialized 14 byte log entry
1026 14:59:29.896653 (+232862us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:29.896742 (+ 89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:29.941743 (+ 45001us) raft_consensus.cc:1685] finished
1026 14:59:29.941772 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:29.941972 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":60831,"prepare.queue_time_us":32171,"raft.queue_time_us":231787,"raft.run_cpu_time_us":572,"raft.run_wall_time_us":48491,"spinlock_wait_cycles":12782336,"thread_start_us":761206,"threads_started":3,"wal-append.queue_time_us":339700}
I20251026 14:59:30.026983 22727 raft_consensus.cc:2410] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251026 14:59:29.994123 25027 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:30.020875 24947 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:30.087544 22728 consensus_queue.cc:260] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.098980 22728 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:30.056872 24593 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:29.936264 25030 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:29.888912 25046 raft_consensus.cc:2749] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:30.095121 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:29.897943 25036 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:30.010283 25045 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:30.094431 24967 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.104224 22734 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:30.109803 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:29.992760 25018 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:29.929068 24877 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:30.115844 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:30.133419 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:30.134238 22805 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:30.018289 24872 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.098451 25026 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.118767 22807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1110) took 1980 ms. Trace:
I20251026 14:59:30.099788 25048 raft_consensus.cc:3060] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:30.120656 22731 raft_consensus.cc:2410] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251026 14:59:30.069875 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:30.137588 22807 rpcz_store.cc:276] 1026 14:59:28.138659 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:28.138821 (+ 162us) service_pool.cc:225] Handling call
1026 14:59:28.751092 (+612271us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:29.282106 (+531014us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:30.073829 (+791723us) spinlock_profiling.cc:243] Waited 517 ms on lock 0x7b540020b7d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:30.118495 (+ 44666us) spinlock_profiling.cc:243] Waited 608 ms on lock 0x7b540020b7d0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:30.118678 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2587665024}
I20251026 14:59:30.142472 22807 raft_consensus.cc:1240] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:30.119262 22729 raft_consensus.cc:1275] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:29.904501 25047 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.158751 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:30.101132 25027 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.159288 22802 raft_consensus.cc:1240] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:30.106101 24947 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:30.162623 25049 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:30.113212 25013 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.120967 25045 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.188133 22733 raft_consensus.cc:2468] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:30.165971 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:30.189090 22728 raft_consensus.cc:1275] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:30.192373 24872 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:30.194648 25026 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:30.196422 24967 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:30.197713 24849 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:30.215721 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:30.216513 22800 raft_consensus.cc:2468] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:30.202625 22728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1005) took 1748 ms. Trace:
I20251026 14:59:30.217394 22728 rpcz_store.cc:276] 1026 14:59:28.453848 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:28.971458 (+517610us) service_pool.cc:225] Handling call
1026 14:59:28.971812 (+ 354us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:30.078065 (+1106253us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b54001cd2a0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d10d5d10 00007fd8d115fd50 00007fd8d1160dc1 00007fd8d115d92f 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c
1026 14:59:30.202032 (+123967us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:30.202558 (+ 526us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":245559680}
I20251026 14:59:30.219717 25047 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:30.227938 25006 consensus_meta.cc:287] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.597s user 0.010s sys 0.000s
I20251026 14:59:30.231213 22733 raft_consensus.cc:1275] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20251026 14:59:30.186622 25049 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:30.215888 24965 consensus_meta.cc:287] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.597s user 0.008s sys 0.002s
I20251026 14:59:30.231901 22800 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714"
dest_uuid: "0d684445287b450a8f738b6ad4b59821"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44570
I20251026 14:59:30.233618 25027 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:30.234549 23402 raft_consensus.cc:1275] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 LEARNER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20251026 14:59:30.237011 24947 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:30.238091 24880 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:30.236917 24873 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:30.246832 24593 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:30.228493 25006 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.248592 22807 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "11ae57d90c614ce2993629f95b777120"
dest_uuid: "0d684445287b450a8f738b6ad4b59821"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:44012
I20251026 14:59:30.278664 22807 raft_consensus.cc:606] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:30.279174 22807 raft_consensus.cc:614] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20251026 14:59:30.250397 22700 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:30.254742 22654 raft_consensus.cc:1240] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:30.257299 22659 raft_consensus.cc:1275] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:30.283620 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:30.284643 22659 raft_consensus.cc:2393] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 6: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:30.258881 22656 raft_consensus.cc:1240] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:30.260247 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:30.260730 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:30.246912 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
W20251026 14:59:30.250305 25005 consensus_meta.cc:287] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.652s user 0.007s sys 0.004s
I20251026 14:59:30.285941 22650 raft_consensus.cc:2468] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:30.287276 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:30.288060 22650 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
W20251026 14:59:30.261162 22697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:30.289811 22697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:30.263131 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:30.291324 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:30.292260 22653 raft_consensus.cc:2393] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 6: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:30.264266 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:30.252813 24964 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:30.266875 22800 raft_consensus.cc:606] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Received request to transfer leadership
W20251026 14:59:30.268589 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 369074us
W20251026 14:59:30.272137 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 370310us
I20251026 14:59:30.276317 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:30.289069 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:30.301682 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:30.303273 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:30.304138 22659 raft_consensus.cc:2468] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:30.306656 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:30.309214 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:30.265501 24965 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.311429 22799 raft_consensus.cc:1240] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:30.305429 25027 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:30.312283 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:30.322149 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:30.317507 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:30.318837 22700 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:30.329911 22803 raft_consensus.cc:1240] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:30.300912 24964 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.336090 25045 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:30.299018 25005 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.319768 25030 consensus_queue.cc:1059] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20251026 14:59:30.322564 25013 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:30.351342 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:30.327670 24947 raft_consensus.cc:2804] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:30.329031 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:30.357389 24947 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:30.324864 25018 consensus_queue.cc:1048] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.031s
W20251026 14:59:30.368791 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 50877.8us
I20251026 14:59:30.364260 22700 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:30.363571 22806 raft_consensus.cc:1240] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:30.372062 22700 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:30.324422 25027 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:30.369616 22624 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:30.377574 22808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1227) took 1440 ms. Trace:
I20251026 14:59:30.380342 22808 rpcz_store.cc:276] 1026 14:59:28.936768 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:28.936941 (+ 173us) service_pool.cc:225] Handling call
1026 14:59:28.949026 (+ 12085us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:29.833033 (+884007us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:30.377181 (+544148us) spinlock_profiling.cc:243] Waited 883 ms on lock 0x7b540038a4d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:30.377470 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2031586176}
I20251026 14:59:30.384394 22700 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:30.336707 25048 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:30.368037 24947 raft_consensus.cc:3060] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:30.377588 24877 raft_consensus.cc:2804] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:30.391908 22700 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:30.384222 24849 raft_consensus.cc:2749] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:30.378260 24945 consensus_queue.cc:1059] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20251026 14:59:30.416496 24945 raft_consensus.cc:3055] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:30.416905 24945 raft_consensus.cc:740] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:30.418025 25049 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:30.362593 25057 consensus_queue.cc:1059] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20251026 14:59:30.419708 25057 raft_consensus.cc:2749] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:30.420331 25057 raft_consensus.cc:3055] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:30.420650 25057 raft_consensus.cc:740] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:30.422160 25005 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:30.424033 25027 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:30.395763 24877 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:30.424928 24877 raft_consensus.cc:3060] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
W20251026 14:59:30.425231 24998 log.cc:927] Time spent T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 1.620s user 0.003s sys 0.006s
I20251026 14:59:30.428552 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1368) took 1815 ms. Trace:
W20251026 14:59:30.430971 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 275166us
I20251026 14:59:30.399892 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:30.383111 24964 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:30.406684 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:30.439821 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:30.429081 22652 rpcz_store.cc:276] 1026 14:59:28.612766 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:28.612982 (+ 216us) service_pool.cc:225] Handling call
1026 14:59:28.613361 (+ 379us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:28.615376 (+ 2015us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:28.615490 (+ 114us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:28.633294 (+ 17804us) log.cc:844] Serialized 26 byte log entry
1026 14:59:28.780493 (+147199us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:28.780533 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:28.780621 (+ 88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:30.424765 (+1644144us) spinlock_profiling.cc:243] Waited 15.3 ms on lock 0x7b24001d8418. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8cdf065a4 00007fd8cdf046ab 00007fd8cdf93409 00007fd8cdf93095 00007fd8cdf926dd 00007fd8d0ce220e 00007fd8d0cf993c 00007fd8d0cf8c66 00007fd8d0cf8888 00007fd8d0d022c1
1026 14:59:30.428149 (+ 3384us) raft_consensus.cc:1685] finished
1026 14:59:30.428176 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:30.428458 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1583,"raft.queue_time_us":1695,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":25045,"spinlock_wait_cycles":35256064,"thread_start_us":117820,"threads_started":1,"wal-append.queue_time_us":167873}
I20251026 14:59:30.477414 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:30.478401 22650 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:30.381368 24831 consensus_queue.cc:1059] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20251026 14:59:30.390718 24759 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:30.479867 22700 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:30.468506 25006 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:30.487025 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 117493us
I20251026 14:59:30.399848 25030 raft_consensus.cc:2749] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:30.500226 22624 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:30.500514 22808 raft_consensus.cc:1240] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:30.501415 22624 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:30.509797 22624 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:30.503821 25039 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.720s user 0.012s sys 0.038s
I20251026 14:59:30.498353 24849 raft_consensus.cc:2804] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:30.528040 22624 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:30.529198 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 59228.8us
I20251026 14:59:30.529956 22774 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:30.530613 22771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 48230.8us
I20251026 14:59:30.530766 22774 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:30.532719 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:30.532914 22774 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:30.533892 22624 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:30.540369 22807 raft_consensus.cc:1240] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:30.503875 24873 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:30.499720 25047 consensus_queue.cc:1059] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20251026 14:59:30.507808 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
W20251026 14:59:30.529091 25039 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.755s user 0.013s sys 0.038s
I20251026 14:59:30.547758 22800 raft_consensus.cc:1240] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:30.505206 24965 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:30.506402 25045 raft_consensus.cc:2749] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:30.549198 25046 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:30.505615 25027 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:30.559769 24831 raft_consensus.cc:2749] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:30.485090 25057 consensus_queue.cc:260] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.574875 24850 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:30.538166 25005 consensus_queue.cc:1048] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20251026 14:59:30.510591 25055 raft_consensus.cc:2749] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:30.566361 22773 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:30.541429 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:30.497179 24759 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.557730 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:59:30.572321 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 95693.9us
I20251026 14:59:30.569382 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:30.562541 22799 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5"
dest_uuid: "0d684445287b450a8f738b6ad4b59821"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:44012
W20251026 14:59:30.552964 25067 log.cc:927] Time spent T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20251026 14:59:30.590237 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 57378.3us
I20251026 14:59:30.590655 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:30.551293 24873 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.546296 24999 raft_consensus.cc:2804] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:30.508492 24885 raft_consensus.cc:2804] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:30.555707 24734 raft_consensus.cc:2749] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:30.556162 25035 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:30.554384 24891 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:30.501988 24945 consensus_queue.cc:260] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.556247 24830 raft_consensus.cc:2749] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251026 14:59:30.555907 24741 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:30.554255 24872 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:30.556041 24967 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:30.549454 25053 raft_consensus.cc:2804] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:30.532436 24849 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:30.596550 24835 raft_consensus.cc:2804] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:30.658005 24835 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:30.577411 24948 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:30.658599 24835 raft_consensus.cc:3060] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:30.614431 22773 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:30.638602 25039 raft_consensus.cc:3055] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:30.611550 24850 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.668823 24850 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:30.624095 22697 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:30.626912 24768 consensus_queue.cc:1059] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
W20251026 14:59:30.670078 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 46091.1us
I20251026 14:59:30.670240 24768 raft_consensus.cc:3055] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:30.670698 24768 raft_consensus.cc:740] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:30.623657 24999 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:30.602246 25057 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:30.674683 24949 raft_consensus.cc:2749] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:30.674805 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:30.626698 25035 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.619597 24841 raft_consensus.cc:2804] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:30.637540 22774 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:30.626791 25066 log.cc:927] Time spent T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20251026 14:59:30.681635 24841 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:30.663509 25039 raft_consensus.cc:740] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:30.682384 24885 raft_consensus.cc:697] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:30.682658 25005 raft_consensus.cc:2749] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251026 14:59:30.600461 25027 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:30.682164 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 44732.3us
I20251026 14:59:30.649575 25048 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:30.691308 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:30.649631 24741 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.651871 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:30.652012 24877 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.653771 25058 raft_consensus.cc:3055] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:30.612164 22799 raft_consensus.cc:606] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:30.694898 22799 raft_consensus.cc:614] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20251026 14:59:30.660924 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 46602.2us
I20251026 14:59:30.695454 22806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1236) took 1091 ms. Trace:
I20251026 14:59:30.695869 22773 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:30.618803 25074 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:30.659017 24948 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.546692 24966 consensus_queue.cc:1048] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20251026 14:59:30.638394 24891 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:30.718967 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 112679us
W20251026 14:59:30.664278 25069 log.cc:927] Time spent T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20251026 14:59:30.659106 24849 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:30.666137 25053 raft_consensus.cc:697] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:30.655560 24872 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.600173 25046 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.674142 22658 raft_consensus.cc:1240] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20251026 14:59:30.671555 24768 consensus_queue.cc:260] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:30.658476 24873 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:30.678174 25035 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:30.682834 25039 consensus_queue.cc:260] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.679955 24759 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:30.683316 24885 consensus_queue.cc:237] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.673949 24745 raft_consensus.cc:2749] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:30.674853 24999 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:30.688220 25027 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:30.688838 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:30.659639 24967 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.695811 22806 rpcz_store.cc:276] 1026 14:59:29.604507 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:29.609779 (+ 5272us) service_pool.cc:225] Handling call
1026 14:59:29.610049 (+ 270us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:30.369724 (+759675us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:30.695243 (+325519us) spinlock_profiling.cc:243] Waited 753 ms on lock 0x7b54000c6554. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:30.695398 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1732281088}
I20251026 14:59:30.701241 22697 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:30.682101 24841 raft_consensus.cc:3060] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:30.711333 24741 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:30.716367 22773 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:30.723318 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:30.729492 22725 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:30.758033 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:30.758272 22800 raft_consensus.cc:1275] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20251026 14:59:30.748178 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:30.758766 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:30.766332 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 65183.6us
I20251026 14:59:30.744737 24877 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:30.807325 22801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1213) took 2311 ms. Trace:
I20251026 14:59:30.718616 25058 raft_consensus.cc:740] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:30.816256 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:30.820327 22732 raft_consensus.cc:3060] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:30.815824 22801 rpcz_store.cc:276] 1026 14:59:28.495873 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:28.791532 (+295659us) service_pool.cc:225] Handling call
1026 14:59:28.791813 (+ 281us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:29.140291 (+348478us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:30.806978 (+1666687us) spinlock_profiling.cc:243] Waited 348 ms on lock 0x7b54001b80d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:30.807237 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":799634688}
I20251026 14:59:30.813685 25075 consensus_queue.cc:1059] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20251026 14:59:30.818259 24946 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:30.743571 24891 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:30.768430 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:30.823067 22807 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:30.823544 22800 pending_rounds.cc:85] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821: Aborting all ops after (but not including) 3
I20251026 14:59:30.744542 24948 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:30.824121 22697 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:30.729738 25036 raft_consensus.cc:493] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
W20251026 14:59:30.828734 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 74718.5us
I20251026 14:59:30.720975 25005 raft_consensus.cc:2749] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251026 14:59:30.831682 22806 raft_consensus.cc:2393] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:30.843506 22800 pending_rounds.cc:107] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20251026 14:59:30.807643 24850 raft_consensus.cc:2749] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:30.814752 25039 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:30.815119 24873 raft_consensus.cc:2749] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:30.845979 22697 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:30.876185 22697 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:30.832388 22655 raft_consensus.cc:3060] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:30.838528 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:30.878376 22807 raft_consensus.cc:2410] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20251026 14:59:30.834103 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:30.847362 22800 raft_consensus.cc:2889] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:30.848116 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:30.769964 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 53691.2us
I20251026 14:59:30.881711 22773 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:30.851750 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:30.872884 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:30.886816 22697 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:30.729092 24843 consensus_queue.cc:1059] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.108s
W20251026 14:59:30.762787 25068 log.cc:927] Time spent T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.156s user 0.000s sys 0.001s
I20251026 14:59:30.969756 22799 raft_consensus.cc:1275] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:30.851775 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:30.972222 22806 raft_consensus.cc:2468] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
W20251026 14:59:30.938544 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 51828.9us
I20251026 14:59:30.979110 22697 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:30.982271 22732 raft_consensus.cc:2468] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:30.983603 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:30.980860 22725 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:31.001587 22697 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:31.002702 22728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1378) took 1075 ms. Trace:
I20251026 14:59:30.834626 24768 raft_consensus.cc:3060] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:31.007593 22773 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:31.013837 22801 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb"
dest_uuid: "0d684445287b450a8f738b6ad4b59821"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:44012
I20251026 14:59:31.014613 22801 raft_consensus.cc:606] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:31.014966 22801 raft_consensus.cc:614] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251026 14:59:30.728508 25074 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:31.003273 22728 rpcz_store.cc:276] 1026 14:59:29.927348 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:30.221726 (+294378us) service_pool.cc:225] Handling call
1026 14:59:30.230831 (+ 9105us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:30.263314 (+ 32483us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:30.263362 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:30.269103 (+ 5741us) log.cc:844] Serialized 26 byte log entry
1026 14:59:30.393000 (+123897us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:30.579165 (+186165us) log.cc:844] Serialized 14 byte log entry
1026 14:59:30.579401 (+ 236us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:30.579455 (+ 54us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:31.002473 (+423018us) raft_consensus.cc:1685] finished
1026 14:59:31.002495 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:31.002636 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":84138,"prepare.queue_time_us":166784,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":188,"raft.run_cpu_time_us":366,"raft.run_wall_time_us":18658,"thread_start_us":182699,"threads_started":2,"wal-append.queue_time_us":224082}
I20251026 14:59:30.866245 24948 raft_consensus.cc:2749] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:30.975541 24727 raft_consensus.cc:2804] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251026 14:59:31.006395 23370 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:30.967374 24745 raft_consensus.cc:2749] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251026 14:59:31.006407 25072 log.cc:927] Time spent T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.173s user 0.000s sys 0.000s
I20251026 14:59:31.000389 24895 consensus_queue.cc:1048] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20251026 14:59:31.027770 24740 raft_consensus.cc:2804] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:31.074936 25018 raft_consensus.cc:2804] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:30.966233 25036 raft_consensus.cc:515] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:30.976683 25049 raft_consensus.cc:2764] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20251026 14:59:30.976074 25026 raft_consensus.cc:2749] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:30.729877 24835 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:30.831661 24946 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:31.057363 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:31.007548 25030 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:31.007483 24850 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:31.031978 24727 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:59:31.090634 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:31.026676 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:31.238710 22732 raft_consensus.cc:3060] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:31.132881 22655 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
W20251026 14:59:31.213420 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 174662us
W20251026 14:59:31.227248 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:31.230502 22801 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2"
dest_uuid: "0d684445287b450a8f738b6ad4b59821"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:44012
I20251026 14:59:31.251276 22700 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:31.241246 22801 raft_consensus.cc:606] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:31.067426 24759 consensus_queue.cc:1059] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.166s
I20251026 14:59:31.010243 25058 consensus_queue.cc:260] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:31.214274 25069 log.cc:927] Time spent T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.149s user 0.001s sys 0.000s
I20251026 14:59:31.244540 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:31.083083 24948 raft_consensus.cc:2749] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251026 14:59:31.260679 22623 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.008s sys 0.035s
W20251026 14:59:31.346853 22623 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.799s user 0.009s sys 0.036s
W20251026 14:59:31.096114 24947 consensus_meta.cc:287] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.697s user 0.011s sys 0.000s
I20251026 14:59:31.019040 25038 consensus_queue.cc:1059] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.209s
W20251026 14:59:31.240509 25072 log.cc:927] Time spent T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.164s user 0.000s sys 0.001s
I20251026 14:59:31.087940 25018 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:31.248792 24727 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
W20251026 14:59:31.358071 22730 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.754s user 0.084s sys 0.160s
W20251026 14:59:31.372584 22730 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.772s user 0.084s sys 0.161s
I20251026 14:59:31.089324 24872 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:31.260071 25057 consensus_meta.cc:287] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.576s user 0.010s sys 0.000s
I20251026 14:59:31.234244 24877 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:31.376047 22540 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 498297us
I20251026 14:59:31.375928 24877 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:31.378329 24877 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:30.820439 25027 consensus_queue.cc:1175] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" responder_term: 8 status { last_received { term: 5 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: [7.4-7.4]" } } last_received_current_leader { term: 0 index: 0 } }
W20251026 14:59:31.375880 25011 log.cc:927] Time spent T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 1.965s user 0.009s sys 0.003s
I20251026 14:59:31.364424 24966 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:31.238679 24740 raft_consensus.cc:697] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:31.382344 24966 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:31.383467 24740 consensus_queue.cc:237] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:31.006768 25053 consensus_queue.cc:237] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:31.371578 24949 raft_consensus.cc:2804] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:31.385497 24949 raft_consensus.cc:697] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:31.375027 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:31.384719 24966 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:31.393143 22800 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d"
dest_uuid: "0d684445287b450a8f738b6ad4b59821"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:44012
I20251026 14:59:31.393818 22800 raft_consensus.cc:606] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:31.394174 22800 raft_consensus.cc:614] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20251026 14:59:31.395000 22727 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.818s user 0.071s sys 0.123s
W20251026 14:59:31.395534 22727 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.818s user 0.071s sys 0.123s
I20251026 14:59:31.398284 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:31.400151 22730 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:31.370699 24967 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:31.250376 25030 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:31.260250 24850 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:31.366349 25018 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:31.235204 24748 consensus_queue.cc:1059] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.130s
W20251026 14:59:31.415102 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:31.417832 22733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1379) took 1490 ms. Trace:
I20251026 14:59:31.355621 22732 raft_consensus.cc:2468] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
W20251026 14:59:31.397821 22624 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.863s user 0.030s sys 0.013s
W20251026 14:59:31.422849 22624 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.888s user 0.030s sys 0.014s
I20251026 14:59:31.423523 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:31.396451 24948 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:31.424320 22732 raft_consensus.cc:2468] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
W20251026 14:59:31.426509 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 892704us
I20251026 14:59:31.428848 22803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1234) took 1824 ms. Trace:
I20251026 14:59:31.429248 22803 rpcz_store.cc:276] 1026 14:59:29.603935 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:29.607298 (+ 3363us) service_pool.cc:225] Handling call
1026 14:59:29.607586 (+ 288us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:30.333215 (+725629us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:31.428597 (+1095382us) spinlock_profiling.cc:243] Waited 722 ms on lock 0x7b540003b1d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:31.428783 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1660576000}
I20251026 14:59:31.420207 22733 rpcz_store.cc:276] 1026 14:59:29.927599 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:30.268076 (+340477us) service_pool.cc:225] Handling call
1026 14:59:30.282600 (+ 14524us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:30.299421 (+ 16821us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:30.299476 (+ 55us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:30.312634 (+ 13158us) log.cc:844] Serialized 52 byte log entry
1026 14:59:30.579154 (+266520us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:30.619430 (+ 40276us) log.cc:844] Serialized 14 byte log entry
1026 14:59:30.623489 (+ 4059us) log.cc:844] Serialized 14 byte log entry
1026 14:59:30.623714 (+ 225us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:30.623814 (+ 100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:31.417605 (+793791us) raft_consensus.cc:1685] finished
1026 14:59:31.417626 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:31.417771 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17832,"prepare.queue_time_us":42889,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":1870,"raft.queue_time_us":25577,"raft.run_cpu_time_us":430,"raft.run_wall_time_us":11125,"thread_start_us":241556,"threads_started":1,"wal-append.queue_time_us":384437}
I20251026 14:59:31.372850 24947 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:31.399693 25046 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:31.457940 22803 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714"
dest_uuid: "0d684445287b450a8f738b6ad4b59821"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44570
I20251026 14:59:31.458843 22803 raft_consensus.cc:606] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Received request to transfer leadership
I20251026 14:59:31.462435 22624 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:31.464673 22604 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 2efb26bccebf43b5b8a85f51fd977714 already in progress
W20251026 14:59:31.465091 22604 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 2efb26bccebf43b5b8a85f51fd977714 already in progress
I20251026 14:59:31.465803 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:31.466657 22733 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:31.472123 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:31.474609 22732 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
W20251026 14:59:31.487152 25048 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.796s user 0.002s sys 0.022s
W20251026 14:59:31.487717 25048 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.796s user 0.002s sys 0.022s
I20251026 14:59:31.490403 22732 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:31.447466 25049 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:31.470772 24748 raft_consensus.cc:3055] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:31.492333 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:31.497545 22726 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.038s sys 0.060s
W20251026 14:59:31.505383 22726 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.562s user 0.039s sys 0.060s
I20251026 14:59:31.505653 22733 raft_consensus.cc:2468] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:31.506139 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1369) took 1611 ms. Trace:
I20251026 14:59:31.443434 24948 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:31.506518 22726 rpcz_store.cc:276] 1026 14:59:29.894511 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:30.105039 (+210528us) service_pool.cc:225] Handling call
1026 14:59:30.105354 (+ 315us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:30.113140 (+ 7786us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:30.113172 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:30.120961 (+ 7789us) log.cc:844] Serialized 26 byte log entry
1026 14:59:30.394744 (+273783us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:30.935113 (+540369us) log.cc:844] Serialized 14 byte log entry
1026 14:59:31.505780 (+570667us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:31.505854 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:31.505901 (+ 47us) raft_consensus.cc:1685] finished
1026 14:59:31.505914 (+ 13us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:31.506081 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7793,"prepare.queue_time_us":135604,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":2906,"raft.queue_time_us":124685,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":28881,"thread_start_us":1234580,"threads_started":3,"wal-append.queue_time_us":326586}
I20251026 14:59:31.522336 22726 raft_consensus.cc:1240] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:31.505108 24748 raft_consensus.cc:740] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:59:31.529738 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 273058us
I20251026 14:59:31.507362 24830 consensus_queue.cc:1048] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20251026 14:59:31.468082 24949 consensus_queue.cc:237] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:31.506793 25049 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:31.534230 22804 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.938s user 0.032s sys 0.034s
W20251026 14:59:31.538407 22804 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.945s user 0.032s sys 0.034s
I20251026 14:59:31.523116 25036 leader_election.cc:290] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:31.540434 22730 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:31.527690 25052 raft_consensus.cc:2749] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:31.531659 25074 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:31.531219 24946 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:31.543926 22699 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.025s sys 0.040s
I20251026 14:59:31.492102 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:31.560246 22699 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.619s user 0.025s sys 0.040s
I20251026 14:59:31.536137 24835 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:31.561403 23370 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:31.563664 22734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 1383) took 1635 ms (client timeout 1861 ms). Trace:
W20251026 14:59:31.564149 22734 rpcz_store.cc:269] 1026 14:59:29.928733 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:30.053610 (+124877us) service_pool.cc:225] Handling call
1026 14:59:31.563475 (+1509865us) spinlock_profiling.cc:243] Waited 46 ms on lock 0x7b54001e9bd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:31.563591 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":105691904}
I20251026 14:59:31.564435 22773 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:31.567772 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:31.568763 24873 raft_consensus.cc:2804] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:31.569423 24873 raft_consensus.cc:697] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:31.539259 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:31.573866 22773 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:31.526515 25086 consensus_queue.cc:1048] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
W20251026 14:59:31.571609 25082 log.cc:927] Time spent T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.202s user 0.000s sys 0.001s
W20251026 14:59:31.585471 24730 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
W20251026 14:59:31.586480 22697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.005s sys 0.018s
W20251026 14:59:31.598600 22697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.596s user 0.005s sys 0.018s
W20251026 14:59:31.599310 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 597753us
I20251026 14:59:31.600253 22697 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:31.601413 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:31.602306 22734 raft_consensus.cc:2468] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:31.604245 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:31.605347 22734 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:31.607120 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:31.578851 25074 raft_consensus.cc:3055] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:31.578523 24964 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:31.615957 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:31.618989 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:31.620095 22730 raft_consensus.cc:3060] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:31.623791 22773 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:31.624792 22773 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:31.570422 24873 consensus_queue.cc:237] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:31.588035 24947 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:31.591413 25052 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:31.588259 25029 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:31.590821 24781 raft_consensus.cc:2804] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251026 14:59:31.631101 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 156974us
I20251026 14:59:31.617587 25074 raft_consensus.cc:740] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:31.621848 24625 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:31.627569 25098 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:31.653373 25098 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:31.654935 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:31.655550 25098 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:59:31.608180 24850 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:31.583132 24748 consensus_queue.cc:260] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:31.660992 25078 log.cc:927] Time spent T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 1.060s user 0.001s sys 0.010s
I20251026 14:59:31.630331 24967 raft_consensus.cc:2804] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:31.661701 24967 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:31.662252 24967 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:31.663811 24734 raft_consensus.cc:3055] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:31.664304 24734 raft_consensus.cc:740] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:31.665171 24734 consensus_queue.cc:260] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:31.665787 22731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1381) took 1737 ms. Trace:
I20251026 14:59:31.641062 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:31.635903 24966 raft_consensus.cc:2804] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:31.667819 24966 raft_consensus.cc:697] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:31.668738 24966 consensus_queue.cc:237] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:31.642490 25029 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:31.598692 25027 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:31.638485 24781 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:31.638103 25094 raft_consensus.cc:2749] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:31.655171 24806 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:31.653788 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:31.641153 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:31.626117 25030 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:31.666204 22731 rpcz_store.cc:276] 1026 14:59:29.928168 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:30.413550 (+485382us) service_pool.cc:225] Handling call
1026 14:59:30.413957 (+ 407us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:30.414349 (+ 392us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:30.414376 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:30.414867 (+ 491us) log.cc:844] Serialized 26 byte log entry
1026 14:59:30.485768 (+ 70901us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:30.524693 (+ 38925us) log.cc:844] Serialized 14 byte log entry
1026 14:59:30.532422 (+ 7729us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:30.532523 (+ 101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:31.660368 (+1127845us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x7b440002304c. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8cdfc0905 00007fd8cdfc1322 00007fd8d0d0a231 00007fd8d0d05b5a 00007fd8d0cf991e 00007fd8d0cf8c66 00007fd8d0cf8888 00007fd8d0d022c1 00007fd8d0d02279 00007fd8d0d02209
1026 14:59:31.665385 (+ 5017us) raft_consensus.cc:1685] finished
1026 14:59:31.665515 (+ 130us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:31.665708 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30596,"prepare.queue_time_us":66233,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":28,"raft.queue_time_us":6844,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":11403,"spinlock_wait_cycles":51194752,"thread_start_us":70462,"threads_started":1,"wal-append.queue_time_us":160115}
W20251026 14:59:31.684980 25093 log.cc:927] Time spent T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
W20251026 14:59:31.685845 25084 log.cc:927] Time spent T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.144s user 0.000s sys 0.001s
I20251026 14:59:31.686977 25096 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:31.688580 25087 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:31.689435 25045 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:31.648851 25091 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:31.700747 22802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1242) took 1890 ms. Trace:
I20251026 14:59:31.628489 24964 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:31.680229 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:31.633138 22773 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:31.937670 22802 rpcz_store.cc:276] 1026 14:59:29.810390 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:29.816825 (+ 6435us) service_pool.cc:225] Handling call
1026 14:59:29.817132 (+ 307us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:30.180554 (+363422us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:31.700462 (+1519908us) spinlock_profiling.cc:243] Waited 330 ms on lock 0x7b540004bad4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:31.700675 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":758866176}
I20251026 14:59:31.937798 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:31.938592 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 306710us
I20251026 14:59:31.936297 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:31.940259 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:31.947937 22730 raft_consensus.cc:2468] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:31.948405 22773 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:31.949679 22773 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:31.952883 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1158) took 1515 ms. Trace:
I20251026 14:59:31.953440 22651 rpcz_store.cc:276] 1026 14:59:30.437047 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:30.437241 (+ 194us) service_pool.cc:225] Handling call
1026 14:59:30.437755 (+ 514us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:30.438267 (+ 512us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:30.438309 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:30.498762 (+ 60453us) log.cc:844] Serialized 52 byte log entry
1026 14:59:30.678306 (+179544us) spinlock_profiling.cc:243] Waited 1.6 ms on lock 0x7b48000801d8. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d606b971 00007fd8d6062a37 00007fd8d60628ca 00007fd8d6159a0f 00007fd8d61596d7 00007fd8d616dcfc 00007fd8d616dca8 00007fd8d616dc28 00007fd8d616dbe3 00007fd8d616ce8c 00007fd8d50aa326
1026 14:59:30.828225 (+149919us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:30.828269 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:30.828367 (+ 98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:31.931056 (+1102689us) raft_consensus.cc:1685] finished
1026 14:59:31.931089 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:31.952775 (+ 21686us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":137,"raft.run_cpu_time_us":824,"raft.run_wall_time_us":207943,"spinlock_wait_cycles":3668736,"thread_start_us":299323,"threads_started":1,"wal-append.queue_time_us":580086}
I20251026 14:59:31.954840 22773 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:31.959422 22773 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:32.003376 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:32.005553 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:32.031059 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:32.032228 22730 raft_consensus.cc:2393] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 6: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:32.046983 22655 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
W20251026 14:59:31.929847 25039 consensus_meta.cc:287] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.956s user 0.006s sys 0.007s
W20251026 14:59:32.055779 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 96456.7us
I20251026 14:59:32.060593 22731 raft_consensus.cc:1240] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:32.064827 22731 raft_consensus.cc:1240] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: []
W20251026 14:59:31.833570 24999 consensus_meta.cc:287] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 1.071s user 0.009s sys 0.003s
W20251026 14:59:31.670979 24769 consensus_queue.cc:1175] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "562bb1e1c8e445b8baecd29c3669658e" responder_term: 5 status { last_received { term: 5 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 5 index: 3 } }
I20251026 14:59:32.115505 22650 raft_consensus.cc:1240] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 8. Current term is 9. Ops: []
I20251026 14:59:32.115736 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:59:31.818382 24841 consensus_meta.cc:287] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 1.003s user 0.013s sys 0.000s
I20251026 14:59:31.703131 24781 raft_consensus.cc:3060] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:32.054739 24745 raft_consensus.cc:2804] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251026 14:59:31.927270 24727 consensus_meta.cc:287] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.526s user 0.010s sys 0.000s
W20251026 14:59:31.730943 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 48906us
I20251026 14:59:32.103155 22651 raft_consensus.cc:1275] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20251026 14:59:31.936941 24748 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:31.747030 24948 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:31.920507 25095 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:31.816028 24855 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:31.883842 25029 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:32.121141 22734 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
W20251026 14:59:32.178938 22771 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.2.4:43341 timed out after 1.527s (SENT) [suppressed 17 similar messages]
I20251026 14:59:31.949412 25049 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:32.221572 22697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.073s sys 0.074s
I20251026 14:59:31.638196 25052 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:31.947182 24946 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:31.892557 24734 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:32.130941 24841 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:31.804529 25074 consensus_queue.cc:260] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:32.114599 24999 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:31.952425 25091 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:31.968732 25096 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:32.101349 24895 raft_consensus.cc:2804] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:32.300055 22656 raft_consensus.cc:1240] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:32.116710 25048 raft_consensus.cc:2804] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:32.099018 25046 raft_consensus.cc:2804] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:32.116526 25098 raft_consensus.cc:2804] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:31.682376 24625 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:32.197618 24745 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:32.197795 25030 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:31.702512 25027 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:32.319619 22652 raft_consensus.cc:1240] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20251026 14:59:32.121937 25094 raft_consensus.cc:493] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:31.976783 25045 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:31.977604 25087 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:32.102350 24849 consensus_meta.cc:287] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 1.345s user 0.007s sys 0.005s
I20251026 14:59:32.374388 24885 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:33.669468 24906 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:33.644846 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:33.670575 24906 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:33.672115 22650 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1402) took 1544 ms (client timeout 1754 ms). Trace:
W20251026 14:59:32.870811 22699 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.017s sys 0.039s
W20251026 14:59:33.673454 22699 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.380s user 0.019s sys 0.054s
I20251026 14:59:33.674539 24873 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:33.675119 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1403) took 1540 ms. Trace:
I20251026 14:59:33.675714 24873 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:32.333014 24948 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:33.678745 24948 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:33.670023 24885 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:33.672830 22650 rpcz_store.cc:269] 1026 14:59:32.128195 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:33.644076 (+1515881us) service_pool.cc:225] Handling call
1026 14:59:33.671978 (+ 27902us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:32.333678 25093 log.cc:927] Time spent T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.131s user 0.000s sys 0.000s
I20251026 14:59:33.675959 22652 rpcz_store.cc:276] 1026 14:59:32.134717 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:32.136252 (+ 1535us) service_pool.cc:225] Handling call
1026 14:59:32.319405 (+183153us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:33.674246 (+1354841us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:33.674877 (+ 631us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:33.643028 22653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1404) took 1442 ms. Trace:
I20251026 14:59:33.703645 25096 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:33.706892 22653 rpcz_store.cc:276] 1026 14:59:32.200823 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:32.201446 (+ 623us) service_pool.cc:225] Handling call
1026 14:59:32.201866 (+ 420us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:32.202132 (+ 266us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:32.202171 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:32.202232 (+ 61us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:33.642126 (+1439894us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:33.642357 (+ 231us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:33.642703 (+ 346us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:33.710944 25096 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:59:32.917605 22774 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.001s sys 0.008s
I20251026 14:59:32.265151 24730 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:33.037840 23367 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.675s user 0.002s sys 0.014s
W20251026 14:59:33.097355 23369 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.734s user 0.005s sys 0.005s
W20251026 14:59:32.596419 25107 log.cc:927] Time spent T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1: Append to log took a long time: real 0.177s user 0.000s sys 0.000s
I20251026 14:59:32.294100 24834 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:32.765666 25035 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.002s sys 0.052s
W20251026 14:59:33.097219 23370 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.719s user 0.008s sys 0.000s
W20251026 14:59:33.088986 24967 consensus_meta.cc:287] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 1.426s user 0.008s sys 0.003s
W20251026 14:59:32.330439 25084 log.cc:927] Time spent T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.133s user 0.000s sys 0.000s
W20251026 14:59:32.594218 25104 log.cc:927] Time spent T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.233s user 0.000s sys 0.000s
I20251026 14:59:32.332175 25016 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:59:33.078737 24748 consensus_meta.cc:287] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.749s user 0.005s sys 0.005s
I20251026 14:59:33.854313 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:33.855393 22656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1406) took 1556 ms. Trace:
I20251026 14:59:33.855890 22656 rpcz_store.cc:276] 1026 14:59:32.299087 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:32.299377 (+ 290us) service_pool.cc:225] Handling call
1026 14:59:32.299797 (+ 420us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:33.831928 (+1532131us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:33.855196 (+ 23268us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:33.856928 24885 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:33.857347 25094 raft_consensus.cc:515] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:33.857777 25096 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:33.860025 25096 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:33.862143 25087 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:33.862908 25114 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:33.863688 25114 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:33.868069 25030 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:33.860031 25094 leader_election.cc:290] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:33.870152 25094 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:33.871057 25094 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:33.874009 25094 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:32.334944 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:33.836091 22731 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.233s user 0.000s sys 0.016s
W20251026 14:59:33.874850 22731 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.272s user 0.000s sys 0.017s
I20251026 14:59:33.836470 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1408) took 1401 ms. Trace:
I20251026 14:59:33.837255 25049 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:33.875406 22657 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1405) took 1608 ms (client timeout 1564 ms). Trace:
W20251026 14:59:33.837826 22697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.079s user 0.089s sys 0.101s
I20251026 14:59:33.837888 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:33.840373 22734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 1402) took 2233 ms (client timeout 2159 ms). Trace:
I20251026 14:59:32.994184 24964 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:33.840862 25052 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:32.329936 25097 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:33.842006 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1401) took 1725 ms. Trace:
W20251026 14:59:33.842451 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 1.66987e+06us
W20251026 14:59:33.843164 22725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 1386) took 3139 ms (client timeout 1558 ms). Trace:
I20251026 14:59:33.843353 24625 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:32.292718 24727 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:33.845407 24772 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:33.845618 24730 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:33.846604 24745 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
W20251026 14:59:33.103833 24806 consensus_meta.cc:287] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 1.127s user 0.011s sys 0.000s
W20251026 14:59:33.033130 25018 consensus_meta.cc:287] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 1.599s user 0.006s sys 0.011s
W20251026 14:59:32.801781 24768 consensus_meta.cc:287] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1: Time spent flushing consensus metadata: real 1.780s user 0.009s sys 0.015s
I20251026 14:59:32.450932 25039 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:33.167312 24781 consensus_meta.cc:287] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.847s user 0.008s sys 0.002s
W20251026 14:59:33.855506 22653 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1413) took 1474 ms (client timeout 1647 ms). Trace:
I20251026 14:59:33.874094 25091 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:33.700934 25106 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
W20251026 14:59:33.107627 24949 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.674s user 0.001s sys 0.008s
W20251026 14:59:33.876030 22657 rpcz_store.cc:269] 1026 14:59:32.267350 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:32.268906 (+ 1556us) service_pool.cc:225] Handling call
1026 14:59:33.875161 (+1606255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:33.881031 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:33.883102 23403 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40908 (request call id 122) took 2615 ms. Trace:
I20251026 14:59:33.885342 24873 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:33.885716 25045 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:33.886009 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 2.25281e+06us
I20251026 14:59:33.886329 25046 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:33.889503 25098 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:33.889915 22650 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1409) took 1567 ms (client timeout 2049 ms). Trace:
I20251026 14:59:33.891364 24948 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:33.875466 22659 rpcz_store.cc:276] 1026 14:59:32.322366 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:32.322993 (+ 627us) service_pool.cc:225] Handling call
1026 14:59:32.323386 (+ 393us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:33.647045 (+1323659us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:33.647145 (+ 100us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:33.647246 (+ 101us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:33.647299 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:33.651539 (+ 4240us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:33.723488 (+ 71949us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:33.892859 24906 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:33.892421 24849 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:33.841378 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:33.898648 22808 raft_consensus.cc:1240] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:33.896667 22651 rpcz_store.cc:276] 1026 14:59:32.102024 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:32.102253 (+ 229us) service_pool.cc:225] Handling call
1026 14:59:32.102695 (+ 442us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:32.296050 (+193355us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:33.820415 (+1524365us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:33.876082 25105 log.cc:927] Time spent T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 1.286s user 0.005s sys 0.016s
W20251026 14:59:33.934746 23367 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.468s user 0.002s sys 0.031s
W20251026 14:59:33.906292 23368 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.302s user 0.003s sys 0.012s
W20251026 14:59:33.906901 23370 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.490s user 0.011s sys 0.007s
I20251026 14:59:33.907351 25113 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:33.907948 25048 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:32.866734 25100 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
W20251026 14:59:33.897332 22725 rpcz_store.cc:269] 1026 14:59:30.568967 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:30.569088 (+ 121us) service_pool.cc:225] Handling call
1026 14:59:33.677064 (+3107976us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54000b9ad4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:33.708562 (+ 31498us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":271799552}
W20251026 14:59:33.876948 22734 rpcz_store.cc:269] 1026 14:59:31.606593 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:31.606727 (+ 134us) service_pool.cc:225] Handling call
1026 14:59:33.839894 (+2233167us) spinlock_profiling.cc:243] Waited 513 ms on lock 0x7b5400077454. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:33.840136 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1180211584}
W20251026 14:59:33.918294 22653 rpcz_store.cc:269] 1026 14:59:32.381088 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:33.803376 (+1422288us) service_pool.cc:225] Handling call
1026 14:59:33.855425 (+ 52049us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:33.917660 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:33.844990 24834 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:33.927013 22650 rpcz_store.cc:269] 1026 14:59:32.322698 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:33.837344 (+1514646us) service_pool.cc:225] Handling call
1026 14:59:33.889802 (+ 52458us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:33.875958 25049 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:33.927922 22774 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.458s user 0.001s sys 0.014s
I20251026 14:59:33.914321 24885 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:33.936663 25013 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:33.936939 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1414) took 1555 ms. Trace:
W20251026 14:59:33.937892 24949 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.525s user 0.003s sys 0.014s
I20251026 14:59:32.375763 24855 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:33.937783 25039 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:33.938779 24895 raft_consensus.cc:697] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:33.938887 24873 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:33.939641 24964 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:33.940307 24031 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:33.940889 24730 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:33.969022 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:33.942137 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1407) took 2326 ms. Trace:
I20251026 14:59:33.943442 24781 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:33.944964 24727 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:33.917179 25052 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:33.946084 23369 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.470s user 0.011s sys 0.007s
I20251026 14:59:33.936721 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:33.927731 25046 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:33.926030 25094 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:33.958200 22652 rpcz_store.cc:276] 1026 14:59:32.381453 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:33.853295 (+1471842us) service_pool.cc:225] Handling call
1026 14:59:33.936853 (+ 83558us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:33.934468 25098 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:33.966596 22656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1412) took 1586 ms. Trace:
I20251026 14:59:33.976490 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:33.976758 22656 rpcz_store.cc:276] 1026 14:59:32.380694 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:33.857198 (+1476504us) service_pool.cc:225] Handling call
1026 14:59:33.966526 (+109328us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:33.951742 23368 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.554s user 0.004s sys 0.012s
W20251026 14:59:33.954764 25035 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.648s user 0.007s sys 0.061s
I20251026 14:59:32.359335 25095 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:33.956646 25048 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:33.920655 23403 rpcz_store.cc:276] 1026 14:59:31.267784 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:31.276416 (+ 8632us) service_pool.cc:225] Handling call
1026 14:59:31.431921 (+155505us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:31.490998 (+ 59077us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:31.495908 (+ 4910us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:31.647964 (+152056us) log.cc:844] Serialized 540 byte log entry
1026 14:59:32.053432 (+405468us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:32.053491 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:32.053602 (+ 111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:33.882412 (+1828810us) raft_consensus.cc:1685] finished
1026 14:59:33.882462 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:33.882794 (+ 332us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":350751,"threads_started":1,"wal-append.queue_time_us":525756}
I20251026 14:59:33.984592 25030 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:33.971859 22727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1384) took 3382 ms. Trace:
W20251026 14:59:33.969355 22654 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1407) took 1647 ms (client timeout 2104 ms). Trace:
I20251026 14:59:33.974372 22652 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f"
dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:46598
W20251026 14:59:33.974566 23370 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:33.974764 22651 raft_consensus.cc:2393] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 8: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:33.975083 25097 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:33.975200 25013 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:33.955605 25016 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:33.920416 24946 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:33.949133 24772 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:33.995986 22729 raft_consensus.cc:1275] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20251026 14:59:33.997220 25026 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:33.997417 25039 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:33.997845 25026 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:33.989782 25018 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.000663 24967 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:33.969971 22726 rpcz_store.cc:276] 1026 14:59:31.615061 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:31.618865 (+ 3804us) service_pool.cc:225] Handling call
1026 14:59:31.619390 (+ 525us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:31.619761 (+ 371us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:31.619792 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:31.620703 (+ 911us) log.cc:844] Serialized 540 byte log entry
1026 14:59:31.950864 (+330161us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:31.950904 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:31.950972 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:33.941508 (+1990536us) raft_consensus.cc:1685] finished
1026 14:59:33.941554 (+ 46us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:33.941872 (+ 318us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":329842,"threads_started":1,"wal-append.queue_time_us":828832}
I20251026 14:59:33.987524 25052 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:33.986660 22654 rpcz_store.cc:269] 1026 14:59:32.321928 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:32.322880 (+ 952us) service_pool.cc:225] Handling call
1026 14:59:33.963239 (+1640359us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:33.987674 25113 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:33.995323 24891 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:33.966123 25045 raft_consensus.cc:493] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:33.974494 24885 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:33.990213 22657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1415) took 1608 ms. Trace:
I20251026 14:59:33.977885 25106 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:34.009430 22728 raft_consensus.cc:2393] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 7: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:34.019681 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1048) took 2076 ms. Trace:
I20251026 14:59:34.012862 22657 rpcz_store.cc:276] 1026 14:59:32.390108 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:33.929225 (+1539117us) service_pool.cc:225] Handling call
1026 14:59:33.990145 (+ 60920us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:34.026858 22726 rpcz_store.cc:276] 1026 14:59:31.943204 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:34.018601 (+2075397us) service_pool.cc:225] Handling call
1026 14:59:34.019033 (+ 432us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:34.019288 (+ 255us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:34.019309 (+ 21us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:34.019367 (+ 58us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:34.019398 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:34.019460 (+ 62us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:34.019582 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:34.029336 24891 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.029968 24967 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:34.030381 25049 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:34.030620 24855 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:34.031044 25049 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:34.031805 25049 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.032999 24999 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:34.034044 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1047) took 2091 ms. Trace:
I20251026 14:59:34.034190 22773 leader_election.cc:304] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:34.034386 24727 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:34.035048 24031 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:34.035152 24727 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.036628 22773 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:34.037516 24964 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.038069 22650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1410) took 1698 ms. Trace:
I20251026 14:59:34.021443 22733 raft_consensus.cc:1240] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:34.034467 22725 rpcz_store.cc:276] 1026 14:59:31.942807 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:34.004566 (+2061759us) service_pool.cc:225] Handling call
1026 14:59:34.005249 (+ 683us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:34.033511 (+ 28262us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:34.033559 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:34.033643 (+ 84us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:34.033712 (+ 69us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:34.033825 (+ 113us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:34.033953 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:34.038465 22650 rpcz_store.cc:276] 1026 14:59:32.339724 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:33.963627 (+1623903us) service_pool.cc:225] Handling call
1026 14:59:34.002462 (+ 38835us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:34.037634 (+ 35172us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:34.037678 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:34.037751 (+ 73us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:34.037805 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:34.037878 (+ 73us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:34.037990 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:34.031890 24891 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:34.042078 22653 raft_consensus.cc:2410] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251026 14:59:34.042325 25113 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:34.042790 25097 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:34.019622 25013 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:33.979031 25006 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:34.009248 25119 log.cc:927] Time spent T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.298s user 0.003s sys 0.000s
I20251026 14:59:34.022650 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:59:34.024307 22774 connection.cc:232] RPC call timeout handler was delayed by -1.5042096549999542s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1985s for a response
I20251026 14:59:34.023214 22652 raft_consensus.cc:606] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:34.020093 22731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1043) took 2085 ms. Trace:
W20251026 14:59:34.057667 22771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 1.7763e+06us
I20251026 14:59:34.072299 22656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1411) took 1729 ms. Trace:
I20251026 14:59:33.997864 24849 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:34.044500 22730 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.070s user 0.000s sys 0.023s
W20251026 14:59:34.072960 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 50303.8us
I20251026 14:59:34.011323 24946 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:34.022301 24895 consensus_queue.cc:237] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.010310 24787 raft_consensus.cc:993] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821: : Instructing follower 562bb1e1c8e445b8baecd29c3669658e to start an election
I20251026 14:59:34.082629 22697 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:34.012705 25094 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:34.061314 24964 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:34.645514 22659 raft_consensus.cc:3060] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:34.151809 23400 raft_consensus.cc:2468] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:34.183153 24853 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:34.647006 24853 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.647353 24921 consensus_queue.cc:1048] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251026 14:59:34.647758 25121 consensus_queue.cc:1059] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251026 14:59:34.355300 22658 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:34.649438 25054 raft_consensus.cc:3055] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:34.649636 24853 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:34.172199 24772 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:34.649986 25054 raft_consensus.cc:740] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:34.631222 25026 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:34.650974 22730 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.785s user 0.015s sys 0.054s
I20251026 14:59:34.129390 22652 raft_consensus.cc:614] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251026 14:59:34.651723 22656 rpcz_store.cc:276] 1026 14:59:32.340111 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:33.977467 (+1637356us) service_pool.cc:225] Handling call
1026 14:59:34.068990 (+ 91523us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:34.069386 (+ 396us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:34.069419 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:34.069486 (+ 67us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:34.069540 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:34.069606 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:34.069723 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:34.078585 22728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 1396) took 3136 ms (client timeout 1564 ms). Trace:
I20251026 14:59:34.593323 22555 catalog_manager.cc:5649] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:59:34.260262 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1403) took 2646 ms. Trace:
W20251026 14:59:34.653694 22728 rpcz_store.cc:269] 1026 14:59:30.941898 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:31.021109 (+ 79211us) service_pool.cc:225] Handling call
1026 14:59:34.078218 (+3057109us) spinlock_profiling.cc:243] Waited 2.73 s on lock 0x7b540008ded4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:34.078470 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6279228800}
I20251026 14:59:34.371814 22802 raft_consensus.cc:1240] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:34.081235 22729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1375) took 4177 ms. Trace:
I20251026 14:59:33.985738 22727 rpcz_store.cc:276] 1026 14:59:30.568428 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:30.570488 (+ 2060us) service_pool.cc:225] Handling call
1026 14:59:30.571858 (+ 1370us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:30.576259 (+ 4401us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:30.576306 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:30.576906 (+ 600us) log.cc:844] Serialized 26 byte log entry
1026 14:59:31.396008 (+819102us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:31.398878 (+ 2870us) log.cc:844] Serialized 14 byte log entry
1026 14:59:31.399103 (+ 225us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:31.399201 (+ 98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:31.567688 (+168487us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b1800094120. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce0b1992 00007fd8ce0b3d61 00007fd8ce0b3d19 00007fd8ce0b3ca9 00007fd8ce0b3c71 00007fd8ce0b2f6d 0000000000379254 0000000000379099 00007fd8ce094d1a
1026 14:59:33.904336 (+2336648us) raft_consensus.cc:1685] finished
1026 14:59:33.916225 (+ 11889us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:33.951185 (+ 34960us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33740,"prepare.queue_time_us":2089,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":1801,"raft.queue_time_us":8807,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":13032,"spinlock_wait_cycles":28386176,"thread_start_us":818750,"threads_started":1,"wal-append.queue_time_us":940360}
I20251026 14:59:34.085114 24031 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
W20251026 14:59:34.348500 22773 connection.cc:232] RPC call timeout handler was delayed by -2.0894952569999532s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15580000000000002s for a response
I20251026 14:59:34.410104 22655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1400) took 2353 ms. Trace:
W20251026 14:59:34.665473 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 503915us
I20251026 14:59:34.043632 24781 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:34.663486 22729 rpcz_store.cc:276] 1026 14:59:29.926415 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:30.157498 (+231083us) service_pool.cc:225] Handling call
1026 14:59:30.157932 (+ 434us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:33.996426 (+3838494us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:34.080836 (+ 84410us) spinlock_profiling.cc:243] Waited 3.8 s on lock 0x7b5400076a54. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:34.081140 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8750614912}
I20251026 14:59:34.669442 22655 rpcz_store.cc:276] 1026 14:59:32.056306 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:32.128064 (+ 71758us) service_pool.cc:225] Handling call
1026 14:59:33.642835 (+1514771us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:34.144568 (+501733us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:34.144622 (+ 54us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:34.161734 (+ 17112us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:34.161800 (+ 66us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:34.394881 (+233081us) spinlock_profiling.cc:243] Waited 470 ms on lock 0x7b54001fbb54. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:34.394922 (+ 41us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:34.409863 (+ 14941us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1081531264}
I20251026 14:59:34.677683 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1050) took 2733 ms. Trace:
I20251026 14:59:34.673031 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1046) took 2724 ms. Trace:
I20251026 14:59:34.673262 22653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1417) took 1032 ms. Trace:
I20251026 14:59:34.064038 25027 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:34.066423 24769 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:34.675529 25094 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
W20251026 14:59:34.630090 23370 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:34.543454 25049 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:34.672521 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1049) took 2720 ms. Trace:
I20251026 14:59:34.204216 25095 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:34.160501 24841 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:34.059258 24967 raft_consensus.cc:2804] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:34.651197 25106 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:34.073786 24727 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:34.159256 25089 log.cc:927] Time spent T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 2.462s user 0.038s sys 0.069s
I20251026 14:59:34.031949 25114 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:34.650497 24787 raft_consensus.cc:1081] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Signalling peer 562bb1e1c8e445b8baecd29c3669658e to start an election
I20251026 14:59:34.678165 22725 rpcz_store.cc:276] 1026 14:59:31.944397 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:34.141550 (+2197153us) service_pool.cc:225] Handling call
1026 14:59:34.667150 (+525600us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:34.677038 (+ 9888us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:34.677333 (+ 295us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:34.677419 (+ 86us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:34.677461 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:34.677514 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:34.677628 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:34.126667 25038 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.745s user 0.013s sys 0.024s
I20251026 14:59:34.159404 24885 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:34.081598 25018 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:34.068569 24786 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:34.246415 24835 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
W20251026 14:59:34.706230 25038 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.325s user 0.025s sys 0.035s
I20251026 14:59:34.182533 25016 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:34.678339 22734 rpcz_store.cc:276] 1026 14:59:31.942482 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:34.023220 (+2080738us) service_pool.cc:225] Handling call
1026 14:59:34.172555 (+149335us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:34.666313 (+493758us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:34.666367 (+ 54us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:34.666431 (+ 64us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:34.666481 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:34.666574 (+ 93us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:34.666753 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:34.689076 24851 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:34.246655 25110 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:34.082391 24834 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:34.666458 25054 consensus_queue.cc:260] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.069825 24843 raft_consensus.cc:3055] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:34.660825 22732 rpcz_store.cc:276] 1026 14:59:31.614024 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:31.614184 (+ 160us) service_pool.cc:225] Handling call
1026 14:59:31.628187 (+ 14003us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:31.628476 (+ 289us) raft_consensus.cc:1534] Early marking committed up to index 5
1026 14:59:31.631193 (+ 2717us) log.cc:844] Serialized 14 byte log entry
1026 14:59:33.841711 (+2210518us) spinlock_profiling.cc:243] Waited 1.64 ms on lock 0x7b18004577e0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce0942f7 00007fd8d6e6d219 00007fd8ce0b0c4b 00007fd8ce0aeeef 00007fd8ce0b11b2 00007fd8d0cf82bb 00007fd8d0cfddc3 00007fd8d0cfe2da 00007fd8d11685ba
1026 14:59:34.161116 (+319405us) log.cc:844] Serialized 14 byte log entry
1026 14:59:34.161451 (+ 335us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:34.192400 (+ 30949us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:34.244370 (+ 51970us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:34.248971 (+ 4601us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:34.258591 (+ 9620us) spinlock_profiling.cc:243] Waited 1.79 ms on lock 0x7b18004577e0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce0b1abf 00007fd8ce0b3d61 00007fd8ce0b3d19 00007fd8ce0b3ca9 00007fd8ce0b3c71 00007fd8ce0b2f6d 0000000000379254 0000000000379099 00007fd8ce094d1a
1026 14:59:34.260057 (+ 1466us) inbound_call.cc:173] Queueing success response
1026 14:59:34.411336 (+151279us) spinlock_profiling.cc:243] Waited 5.55 ms on lock 0x7b18004577e0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce0b1992 00007fd8ce0b3d61 00007fd8ce0b3d19 00007fd8ce0b3ca9 00007fd8ce0b3c71 00007fd8ce0b2f6d 0000000000379254 0000000000379099 00007fd8ce094d1a
Metrics: {"mutex_wait_us":12047,"prepare.queue_time_us":2156,"prepare.run_cpu_time_us":121,"prepare.run_wall_time_us":125,"raft.queue_time_us":18743,"raft.run_cpu_time_us":347,"raft.run_wall_time_us":70902,"spinlock_wait_cycles":20653184,"thread_start_us":111555,"threads_started":1,"wal-append.queue_time_us":2375576}
I20251026 14:59:34.259070 25031 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:34.115011 22731 rpcz_store.cc:276] 1026 14:59:31.934776 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:32.066669 (+131893us) service_pool.cc:225] Handling call
1026 14:59:32.067039 (+ 370us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:32.067314 (+ 275us) raft_consensus.cc:1534] Early marking committed up to index 4
1026 14:59:32.603019 (+535705us) log.cc:844] Serialized 14 byte log entry
1026 14:59:34.019249 (+1416230us) spinlock_profiling.cc:243] Waited 4.49 ms on lock 0x7b1800429100. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8d0d0655c 00007fd8ce09497b 00007fd8d6e6d219 00007fd8ce0b0c4b 00007fd8ce0aeeef 00007fd8ce0b11b2 00007fd8d0cf82bb 00007fd8d0cfddc3 00007fd8d0cfe2da
1026 14:59:34.019558 (+ 309us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:34.019630 (+ 72us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:34.019736 (+ 106us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:34.019863 (+ 127us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:34.020001 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":798368,"prepare.run_cpu_time_us":15397,"prepare.run_wall_time_us":747837,"raft.queue_time_us":728,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":195,"spinlock_wait_cycles":10332544,"thread_start_us":1271989,"threads_started":1,"wal-append.queue_time_us":1361662}
I20251026 14:59:34.057904 24999 raft_consensus.cc:2749] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:34.693858 22726 rpcz_store.cc:276] 1026 14:59:31.943821 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:34.027957 (+2084136us) service_pool.cc:225] Handling call
1026 14:59:34.141649 (+113692us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:34.663913 (+522264us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:34.663931 (+ 18us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:34.663963 (+ 32us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:34.663988 (+ 25us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:34.664197 (+ 209us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:34.664353 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:34.695530 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:34.020565 25045 raft_consensus.cc:515] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:34.394204 24527 raft_consensus.cc:3055] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:34.189795 25098 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.698004 25026 raft_consensus.cc:2749] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:34.390444 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
W20251026 14:59:34.705101 22773 connection.cc:232] RPC call timeout handler was delayed by -1.698925982999981s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1564s for a response
I20251026 14:59:34.181408 24745 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.116323 24873 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.699020 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:34.181567 25006 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.703356 24837 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:34.709043 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:34.703483 25036 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:34.187903 24880 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:34.188577 25100 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:34.711385 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:34.678470 22653 rpcz_store.cc:276] 1026 14:59:33.644433 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:33.957369 (+312936us) service_pool.cc:225] Handling call
1026 14:59:34.673169 (+715800us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:34.720579 25053 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:34.706583 25114 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:34.711122 24967 raft_consensus.cc:697] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:34.732704 24885 consensus_queue.cc:1059] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20251026 14:59:34.729205 22659 raft_consensus.cc:2468] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:34.704471 24769 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.724256 24843 raft_consensus.cc:740] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:34.764250 24999 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
W20251026 14:59:34.735591 22774 connection.cc:232] RPC call timeout handler was delayed by -1.4214096550000477s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2077s for a response
I20251026 14:59:34.730000 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:34.728758 25054 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
W20251026 14:59:34.731468 24740 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.599s user 0.008s sys 0.025s
W20251026 14:59:34.741648 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 719032us
I20251026 14:59:34.749714 22800 raft_consensus.cc:1240] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:34.740396 22555 catalog_manager.cc:5649] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:34.748142 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:34.756431 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:34.757040 22725 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:34.735880 24527 raft_consensus.cc:740] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:34.759800 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:34.714800 25052 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:34.757463 22728 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:34.753602 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:34.754340 24880 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } }
I20251026 14:59:34.738374 24851 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.723896 22624 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:34.766718 22659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1416) took 1971 ms (client timeout 2218 ms). Trace:
W20251026 14:59:34.781145 22659 rpcz_store.cc:269] 1026 14:59:32.795420 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:33.948940 (+1153520us) service_pool.cc:225] Handling call
1026 14:59:34.766609 (+817669us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":365568}
I20251026 14:59:34.767624 22727 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
W20251026 14:59:34.768719 22774 connection.cc:232] RPC call timeout handler was delayed by -1.3769209820000015s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2049s for a response
I20251026 14:59:34.758457 24837 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.738855 25031 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.738991 25016 consensus_queue.cc:1059] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20251026 14:59:34.766633 25037 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:34.774587 24787 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:34.771112 25036 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:34.766291 25048 consensus_meta.cc:287] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.782s user 0.012s sys 0.000s
I20251026 14:59:34.754683 25110 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.787817 25037 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.748745 24835 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:34.772600 24740 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.712s user 0.008s sys 0.025s
I20251026 14:59:34.780567 22733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1390) took 4210 ms. Trace:
W20251026 14:59:34.782725 22774 connection.cc:232] RPC call timeout handler was delayed by -1.3557570610000766s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21040000000000003s for a response
I20251026 14:59:34.783717 22732 raft_consensus.cc:2468] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:34.778712 22734 raft_consensus.cc:2468] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:34.768234 24745 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:34.785109 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:34.783805 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:34.799125 22774 connection.cc:232] RPC call timeout handler was delayed by -1.317909654999994s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21920000000000003s for a response
I20251026 14:59:34.724217 24786 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.784001 22697 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:34.769320 25114 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.792549 22726 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 3.
I20251026 14:59:34.799885 22729 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:34.786453 24945 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:34.796128 22733 rpcz_store.cc:276] 1026 14:59:30.569858 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:31.511317 (+941459us) service_pool.cc:225] Handling call
1026 14:59:31.524790 (+ 13473us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:34.059911 (+2535121us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:34.780210 (+720299us) spinlock_profiling.cc:243] Waited 2.5 s on lock 0x7b540008d4d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:34.780505 (+ 295us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5740394368}
W20251026 14:59:34.812794 22774 connection.cc:232] RPC call timeout handler was delayed by -1.2781697600000825s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1647s for a response
I20251026 14:59:34.813627 22731 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:34.797096 22555 catalog_manager.cc:5649] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 2 to 4, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 4 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:59:34.778128 25053 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.825644 25015 raft_consensus.cc:2749] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251026 14:59:34.825448 24967 consensus_queue.cc:237] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:34.826380 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 171249us
I20251026 14:59:34.829996 25074 raft_consensus.cc:3060] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:34.793133 24843 consensus_queue.cc:260] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.832583 24843 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:34.833233 25100 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:34.835242 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1052) took 2890 ms. Trace:
I20251026 14:59:34.835698 22734 rpcz_store.cc:276] 1026 14:59:31.945233 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:34.816423 (+2871190us) service_pool.cc:225] Handling call
1026 14:59:34.834663 (+ 18240us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:34.834872 (+ 209us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:34.834895 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:34.834935 (+ 40us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:34.834980 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:34.835041 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:34.835150 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:34.837726 24527 consensus_queue.cc:260] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.839502 24527 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:34.810083 24999 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:34.843493 25098 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:34.844215 24769 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:34.793138 25048 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.848057 25048 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:34.850896 22734 raft_consensus.cc:1240] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:34.846099 22555 catalog_manager.cc:5649] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 3 to 4, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 4 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
W20251026 14:59:34.857175 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 501700us
I20251026 14:59:34.788368 24787 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:34.865864 22771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 51126.5us
I20251026 14:59:34.869026 22806 raft_consensus.cc:1240] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:34.857635 25026 raft_consensus.cc:2749] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251026 14:59:34.892086 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 108209us
I20251026 14:59:34.885850 22624 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:34.895417 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:34.877866 25045 leader_election.cc:290] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:34.889955 25006 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:34.907016 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:34.907246 22624 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:34.908154 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:34.903697 22555 catalog_manager.cc:5649] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 1 to 4, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 4 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:59:34.900744 24835 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:34.913445 22730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1045) took 2971 ms. Trace:
I20251026 14:59:34.919009 25016 raft_consensus.cc:2749] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251026 14:59:34.903155 24873 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:34.922122 22697 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:34.844684 25098 raft_consensus.cc:2955] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Committing config change with OpId 6.6: config changed from index 2 to 6, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:59:34.932876 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:34.933517 25046 consensus_meta.cc:287] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.958s user 0.012s sys 0.004s
I20251026 14:59:34.933970 25046 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:34.935859 22734 raft_consensus.cc:1275] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:34.936131 25046 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:34.936872 25026 raft_consensus.cc:2804] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:34.915447 24786 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:34.937223 22734 raft_consensus.cc:1275] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251026 14:59:34.939855 22555 catalog_manager.cc:5649] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:59:34.916064 22730 rpcz_store.cc:276] 1026 14:59:31.942091 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:32.103300 (+161209us) service_pool.cc:225] Handling call
1026 14:59:32.103648 (+ 348us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:32.103906 (+ 258us) raft_consensus.cc:1534] Early marking committed up to index 4
1026 14:59:34.686629 (+2582723us) log.cc:844] Serialized 14 byte log entry
1026 14:59:34.891176 (+204547us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:34.896102 (+ 4926us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:34.899816 (+ 3714us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:34.904163 (+ 4347us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:34.913384 (+ 9221us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":499319,"prepare.queue_time_us":1794426,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":85,"raft.queue_time_us":4153,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":2786,"thread_start_us":1943301,"threads_started":2,"wal-append.queue_time_us":207896}
I20251026 14:59:34.915447 25110 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:34.941730 22803 raft_consensus.cc:1240] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:34.944875 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:34.947681 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 50274.2us
I20251026 14:59:34.946345 22801 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:34.931833 24880 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:34.952802 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:34.953656 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:34.954617 24845 raft_consensus.cc:2749] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:34.955101 25053 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:34.950562 24999 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:34.956471 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:34.956707 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:34.955438 25058 raft_consensus.cc:3060] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:34.937335 25026 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:34.963555 25036 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:34.971503 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 64390.3us
I20251026 14:59:34.952979 22773 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:34.972784 22727 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:34.973443 22733 raft_consensus.cc:1240] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:34.932862 25045 consensus_queue.cc:1048] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.020s
I20251026 14:59:34.974169 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:34.979558 22806 raft_consensus.cc:2410] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251026 14:59:34.977094 22773 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:34.977577 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:34.948246 24837 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:34.983085 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:34.983563 22773 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:34.984448 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:34.961715 23404 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:34.986162 22730 raft_consensus.cc:2393] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 7: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:34.982008 22734 raft_consensus.cc:2376] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7. Re-sending same reply.
I20251026 14:59:34.989431 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:34.942171 24873 consensus_queue.cc:1048] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.033s
W20251026 14:59:34.990931 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 68903.5us
I20251026 14:59:34.962073 25026 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:34.997370 24849 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:34.990463 24999 raft_consensus.cc:2804] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:34.952773 22555 catalog_manager.cc:5649] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 3 to 4, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1), no longer has a pending config: VOTER df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1), VOTER 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2), VOTER 0d684445287b450a8f738b6ad4b59821 (127.21.2.3), NON_VOTER 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4). New cstate: current_term: 4 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:59:35.001469 25036 raft_consensus.cc:2804] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:35.001943 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:35.002624 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:35.001989 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:35.005674 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:35.006716 22773 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:35.007406 25053 raft_consensus.cc:2804] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:35.009702 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.018570 22773 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:35.018934 25053 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:35.019498 24740 raft_consensus.cc:2804] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:35.019665 24781 raft_consensus.cc:2804] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:35.018465 23399 raft_consensus.cc:2955] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 2 to 6, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:59:35.019956 24740 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:35.020159 22773 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:35.020190 24781 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:35.020426 24740 raft_consensus.cc:3060] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:35.020848 24781 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:34.979991 25114 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:35.022979 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:35.033560 22555 catalog_manager.cc:5649] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:59:35.019205 22697 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:35.042690 22808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1256) took 4445 ms. Trace:
I20251026 14:59:35.043294 22808 rpcz_store.cc:276] 1026 14:59:30.597177 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:30.619439 (+ 22262us) service_pool.cc:225] Handling call
1026 14:59:30.619793 (+ 354us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:33.962145 (+3342352us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:35.042291 (+1080146us) spinlock_profiling.cc:243] Waited 3.27 s on lock 0x7b540020bcd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:35.042579 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7529101696}
I20251026 14:59:35.045789 24837 consensus_queue.cc:1059] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:59:35.046641 24837 consensus_queue.cc:1243] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer 562bb1e1c8e445b8baecd29c3669658e log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20251026 14:59:35.047194 24837 raft_consensus.cc:3055] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:35.046280 22555 catalog_manager.cc:5649] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 reported cstate change: term changed from 2 to 3, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 3 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:59:35.048138 23401 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:35.047479 24837 raft_consensus.cc:740] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:35.045491 22808 raft_consensus.cc:1240] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:35.059584 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.060438 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:35.030031 25036 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:35.019506 25053 raft_consensus.cc:3060] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:35.014456 24999 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:35.060338 22653 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
W20251026 14:59:35.062898 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 87774.9us
I20251026 14:59:35.062948 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.067571 22725 raft_consensus.cc:2468] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:35.040652 24835 raft_consensus.cc:2804] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20251026 14:59:35.072079 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:35.041399 22730 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2"
dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:35092
I20251026 14:59:35.059458 24873 consensus_queue.cc:1243] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer 562bb1e1c8e445b8baecd29c3669658e log is divergent from this leader: its last log entry 8.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20251026 14:59:35.097584 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:59:35.081589 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:35.081852 22806 raft_consensus.cc:1275] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20251026 14:59:35.070133 25036 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:35.082460 22808 raft_consensus.cc:1240] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:35.082280 24849 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.083137 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20251026 14:59:35.083492 24837 consensus_queue.cc:260] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.083904 22802 raft_consensus.cc:1240] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20251026 14:59:35.092346 24895 consensus_queue.cc:1059] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251026 14:59:35.096982 24877 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:35.069728 24999 raft_consensus.cc:3060] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
W20251026 14:59:35.125523 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 50036.9us
I20251026 14:59:35.183326 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:59:35.184688 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20251026 14:59:35.185950 25136 log.cc:927] Time spent T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.171s user 0.000s sys 0.000s
I20251026 14:59:35.186934 24849 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:35.187623 24837 raft_consensus.cc:3060] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:35.189512 22774 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:35.190970 23370 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:35.191434 24877 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.193588 24945 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:35.194509 24945 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:35.197395 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 126666us
I20251026 14:59:35.199162 24853 consensus_queue.cc:1059] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20251026 14:59:35.199476 22624 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:35.200263 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 125284us
I20251026 14:59:35.200434 24853 raft_consensus.cc:3060] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:35.197009 22728 raft_consensus.cc:1275] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20251026 14:59:35.204476 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 185352us
I20251026 14:59:35.072194 24835 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:35.100651 22730 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 14:59:35.057880 25121 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:35.100165 22806 pending_rounds.cc:85] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821: Aborting all ops after (but not including) 4
I20251026 14:59:35.212430 22730 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:35.217630 25121 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.218088 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.219630 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.220013 25121 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:35.221068 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.221661 22807 raft_consensus.cc:1240] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: []
W20251026 14:59:35.222695 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 111211us
I20251026 14:59:35.217964 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.226559 24945 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:35.226490 25046 consensus_queue.cc:1059] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20251026 14:59:35.228190 25046 raft_consensus.cc:3055] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:35.228572 25046 raft_consensus.cc:740] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:35.229050 22659 raft_consensus.cc:1275] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20251026 14:59:35.230147 24841 consensus_queue.cc:1048] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:59:35.231063 22654 raft_consensus.cc:1275] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:35.231241 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.232111 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.232445 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.233224 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.229416 25046 consensus_queue.cc:260] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.234426 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.235119 22654 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:35.235178 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.236318 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.233148 22774 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:35.236027 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.212805 22806 pending_rounds.cc:107] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20251026 14:59:35.252501 22806 raft_consensus.cc:2889] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:35.074805 25045 raft_consensus.cc:2749] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:35.237018 25046 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
W20251026 14:59:35.267728 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 58722.8us
I20251026 14:59:35.277429 22729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1055) took 1235 ms. Trace:
I20251026 14:59:35.270675 22622 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:35.222599 24895 raft_consensus.cc:2749] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251026 14:59:35.277596 22773 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:35.277555 23404 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:35.213965 24835 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:35.284042 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.289392 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.290172 22653 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:35.282037 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:35.111402 25110 consensus_queue.cc:1059] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.156s
I20251026 14:59:35.293727 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:35.296936 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:35.297674 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 80266.2us
I20251026 14:59:35.303712 24741 consensus_queue.cc:1048] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20251026 14:59:35.266960 24853 raft_consensus.cc:2749] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:35.283233 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:35.310310 22730 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.278035 22729 rpcz_store.cc:276] 1026 14:59:34.032677 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:34.812641 (+779964us) service_pool.cc:225] Handling call
1026 14:59:34.829184 (+ 16543us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:34.864226 (+ 35042us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:34.864281 (+ 55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:34.893901 (+ 29620us) log.cc:844] Serialized 26 byte log entry
1026 14:59:35.009571 (+115670us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:35.267151 (+257580us) log.cc:844] Serialized 14 byte log entry
1026 14:59:35.267536 (+ 385us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:35.267649 (+ 113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:35.267707 (+ 58us) raft_consensus.cc:1685] finished
1026 14:59:35.267727 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:35.267919 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5313,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":13280,"thread_start_us":301150,"threads_started":2,"wal-append.queue_time_us":136774}
I20251026 14:59:35.328831 22730 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:35.330399 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.330636 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.331898 22659 raft_consensus.cc:2410] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251026 14:59:35.334316 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:35.332144 25137 raft_consensus.cc:2749] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251026 14:59:35.340152 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 69590.6us
I20251026 14:59:35.340958 22650 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:35.341949 22650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 1169) took 1281 ms. Trace:
I20251026 14:59:35.342322 22650 rpcz_store.cc:276] 1026 14:59:34.060566 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:35.294766 (+1234200us) service_pool.cc:225] Handling call
1026 14:59:35.341873 (+ 47107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:35.345153 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:35.346415 22659 raft_consensus.cc:2393] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 6: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:35.347630 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.348392 22807 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:35.349884 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:35.351845 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.352674 24741 raft_consensus.cc:3055] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 8 LEADER]: Stepping down as leader of term 8
I20251026 14:59:35.353142 22653 raft_consensus.cc:2468] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:35.354669 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:59:35.357754 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 124708us
I20251026 14:59:35.376461 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.353143 24741 raft_consensus.cc:740] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:35.325927 25006 consensus_queue.cc:1048] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20251026 14:59:35.367584 25036 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.363689 24853 raft_consensus.cc:2749] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:35.374977 25114 raft_consensus.cc:2804] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:35.392239 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.394182 22622 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:35.397025 22622 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:35.330392 24740 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.401741 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.402580 22654 raft_consensus.cc:2468] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:35.402001 22622 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:35.400828 22651 raft_consensus.cc:2468] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:35.403748 25114 raft_consensus.cc:697] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:35.412005 25114 consensus_queue.cc:237] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.416087 24835 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.416863 24851 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:35.419744 23400 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40908 (request call id 121) took 4768 ms (client timeout 1527 ms). Trace:
I20251026 14:59:35.418192 22808 raft_consensus.cc:1240] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 7. Current term is 8. Ops: []
W20251026 14:59:35.418725 25140 log.cc:927] Time spent T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20251026 14:59:35.419205 24741 consensus_queue.cc:260] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.426378 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:35.431959 22658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1396) took 3753 ms (client timeout 1806 ms). Trace:
I20251026 14:59:35.406697 25121 raft_consensus.cc:2804] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:35.430944 23403 raft_consensus.cc:1275] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251026 14:59:35.407171 24885 raft_consensus.cc:2749] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251026 14:59:35.432391 22658 rpcz_store.cc:269] 1026 14:59:31.675715 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:31.675911 (+ 196us) service_pool.cc:225] Handling call
1026 14:59:35.420740 (+3744829us) spinlock_profiling.cc:243] Waited 2.26 s on lock 0x7b54001fc554. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:35.428501 (+ 7761us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5208407424}
I20251026 14:59:35.441531 23404 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:35.439514 22774 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:35.448338 22622 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:35.442379 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1054) took 3496 ms. Trace:
I20251026 14:59:35.427803 24853 raft_consensus.cc:2749] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:35.457705 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:35.464001 25134 log.cc:927] Time spent T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.545s user 0.000s sys 0.004s
I20251026 14:59:35.464161 24781 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.465114 23401 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:35.451810 24769 raft_consensus.cc:2749] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:35.466367 24748 consensus_queue.cc:1059] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20251026 14:59:35.467196 24781 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:35.406580 25124 consensus_queue.cc:1059] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20251026 14:59:35.470551 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:35.426041 23400 rpcz_store.cc:269] 1026 14:59:30.651396 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:30.651563 (+ 167us) service_pool.cc:225] Handling call
1026 14:59:35.419429 (+4767866us) spinlock_profiling.cc:243] Waited 3.25 s on lock 0x7b540053e3d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:35.419624 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7468169600}
I20251026 14:59:35.471278 22729 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:35.471760 25041 raft_consensus.cc:3055] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:35.472208 25041 raft_consensus.cc:740] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:35.473500 24527 raft_consensus.cc:3055] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:35.473212 25041 consensus_queue.cc:260] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.473875 24527 raft_consensus.cc:740] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:35.474876 25041 raft_consensus.cc:3060] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:35.474763 24527 consensus_queue.cc:260] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.475346 24967 raft_consensus.cc:2749] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:35.476292 24527 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:35.477372 22730 raft_consensus.cc:1240] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:35.479068 25036 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:35.479552 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.480394 22654 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 9.
I20251026 14:59:35.481666 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.483341 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:35.483912 25053 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.485234 22805 raft_consensus.cc:1240] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:35.485790 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.486647 22655 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:35.487864 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.488690 22657 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:35.491317 25031 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:35.446125 24741 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:35.495487 25037 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:35.503304 24787 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:35.505208 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:35.506268 22731 raft_consensus.cc:3060] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:35.509555 24031 raft_consensus.cc:3055] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:35.509975 24031 raft_consensus.cc:740] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:59:35.510051 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 70623.9us
I20251026 14:59:35.510845 25038 raft_consensus.cc:3055] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:35.511293 25038 raft_consensus.cc:740] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:35.511492 22803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1263) took 1238 ms. Trace:
I20251026 14:59:35.509004 24999 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.512172 25038 consensus_queue.cc:260] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.513334 22804 raft_consensus.cc:1240] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:35.514425 24877 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:35.514516 24873 raft_consensus.cc:3055] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:35.515101 24873 raft_consensus.cc:740] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:35.516007 24873 consensus_queue.cc:260] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.516762 24999 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:35.517578 23403 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40908 (request call id 124) took 1389 ms. Trace:
I20251026 14:59:35.517880 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.518033 23403 rpcz_store.cc:276] 1026 14:59:34.128080 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:34.128366 (+ 286us) service_pool.cc:225] Handling call
1026 14:59:35.425738 (+1297372us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:35.439652 (+ 13914us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:35.517259 (+ 77607us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b540053e3d0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:35.517491 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2970306944}
I20251026 14:59:35.518930 22659 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:35.520159 22733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1057) took 1486 ms. Trace:
I20251026 14:59:35.520588 22733 rpcz_store.cc:276] 1026 14:59:34.039510 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:34.815357 (+775847us) service_pool.cc:225] Handling call
1026 14:59:34.973246 (+157889us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:34.973811 (+ 565us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:35.519933 (+546122us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b540008e8d0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:35.520085 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":291217280}
W20251026 14:59:35.521834 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:35.488695 25053 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:35.435285 25121 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:35.525375 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1053) took 3579 ms. Trace:
I20251026 14:59:35.479107 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.449528 22726 rpcz_store.cc:276] 1026 14:59:31.945928 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:34.817004 (+2871076us) service_pool.cc:225] Handling call
1026 14:59:34.876560 (+ 59556us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:34.893655 (+ 17095us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:34.893681 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:34.904590 (+ 10909us) log.cc:844] Serialized 26 byte log entry
1026 14:59:35.061100 (+156510us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:35.061150 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:35.061222 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:35.442185 (+380963us) raft_consensus.cc:1685] finished
1026 14:59:35.442209 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:35.442325 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":13180,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":5725,"thread_start_us":117064,"threads_started":1,"wal-append.queue_time_us":360259}
I20251026 14:59:35.525851 22732 rpcz_store.cc:276] 1026 14:59:31.945575 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:34.809388 (+2863813us) service_pool.cc:225] Handling call
1026 14:59:34.826827 (+ 17439us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:34.827102 (+ 275us) raft_consensus.cc:1534] Early marking committed up to index 7
1026 14:59:35.039929 (+212827us) log.cc:844] Serialized 14 byte log entry
1026 14:59:35.400390 (+360461us) log.cc:844] Serialized 14 byte log entry
1026 14:59:35.524809 (+124419us) spinlock_profiling.cc:243] Waited 2.21 ms on lock 0x7b18003b4b80. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce026caa 00007fd8d7698a29 00007fd8cdee6c30 00007fd8ce0aea94 00007fd8ce0aea05 00007fd8d1155c00 00007fd8d11682a6 00007fd8d1178e17 00007fd8d1178d98
1026 14:59:35.524978 (+ 169us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:35.525039 (+ 61us) raft_consensus.cc:1645] Marking committed up to 7
1026 14:59:35.525090 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:35.525173 (+ 83us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:35.525315 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7254,"prepare.queue_time_us":394480,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":2946,"raft.queue_time_us":3636,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":9144,"spinlock_wait_cycles":5074688,"thread_start_us":457612,"threads_started":2,"wal-append.queue_time_us":363259}
I20251026 14:59:35.441978 24740 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:35.510847 24031 consensus_queue.cc:260] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.525578 25121 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:35.563238 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.565137 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.566790 24748 raft_consensus.cc:3055] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:35.567261 24748 raft_consensus.cc:740] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:35.568033 25038 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:35.568182 24748 consensus_queue.cc:260] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.562588 25053 consensus_queue.cc:1048] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20251026 14:59:35.481663 25026 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.573724 22727 raft_consensus.cc:2393] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 7: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:35.574119 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.561952 24873 raft_consensus.cc:3060] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:35.574955 22650 raft_consensus.cc:2393] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 9: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:35.577781 24835 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:35.511848 22803 rpcz_store.cc:276] 1026 14:59:34.272576 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:34.287544 (+ 14968us) service_pool.cc:225] Handling call
1026 14:59:34.288124 (+ 580us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:34.942293 (+654169us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:35.511193 (+568900us) spinlock_profiling.cc:243] Waited 649 ms on lock 0x7b540020c6d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:35.511399 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1493023616}
I20251026 14:59:35.546370 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.545423 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.584400 22658 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
W20251026 14:59:35.603317 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 325754us
I20251026 14:59:35.604588 22773 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:35.582306 25058 consensus_meta.cc:287] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.606s user 0.011s sys 0.000s
I20251026 14:59:35.624950 22725 raft_consensus.cc:1240] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:35.628140 22622 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:35.561712 24853 consensus_queue.cc:1059] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20251026 14:59:35.636883 22732 raft_consensus.cc:1240] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:35.634014 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:59:35.641089 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:35.642199 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:35.642385 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:35.643088 22726 raft_consensus.cc:2410] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 10 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251026 14:59:35.643944 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:35.645201 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 235727us
I20251026 14:59:35.646986 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.647301 22729 raft_consensus.cc:2410] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251026 14:59:35.649502 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:35.650465 22725 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:35.630195 24740 raft_consensus.cc:2804] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:35.615394 24769 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:35.652398 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:35.634771 25074 raft_consensus.cc:2804] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:35.652503 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:35.607570 24851 consensus_queue.cc:1059] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20251026 14:59:35.654606 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:35.655078 22659 raft_consensus.cc:2393] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 8: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:35.656680 22773 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:35.660588 24748 raft_consensus.cc:3060] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:35.663041 24851 raft_consensus.cc:3055] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:35.663470 24851 raft_consensus.cc:740] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:35.664439 24851 consensus_queue.cc:260] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.665825 23401 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:35.656682 24332 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:35.666878 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:35.668469 22727 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:35.670397 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:35.671332 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:35.672084 22732 raft_consensus.cc:2410] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20251026 14:59:35.634475 24527 consensus_queue.cc:1048] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20251026 14:59:35.656872 25124 consensus_queue.cc:1059] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20251026 14:59:35.691582 25114 raft_consensus.cc:2804] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:35.692328 24740 raft_consensus.cc:697] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:35.693994 24835 consensus_queue.cc:1048] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251026 14:59:35.672681 25045 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:35.695801 25045 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.696373 22624 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:35.693215 24740 consensus_queue.cc:237] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.701454 24843 raft_consensus.cc:3055] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:35.701892 24843 raft_consensus.cc:740] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:35.701913 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.702776 24843 consensus_queue.cc:260] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.703303 22655 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:35.704200 24843 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:35.684372 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:35.706013 22728 raft_consensus.cc:2410] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251026 14:59:35.685863 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:35.677063 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.687620 24877 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:35.674981 24332 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.688298 24769 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.714553 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:35.688789 23401 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:35.668241 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:35.667629 25074 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:35.727591 25054 raft_consensus.cc:2804] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:35.705188 25058 consensus_queue.cc:1059] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20251026 14:59:35.728132 25054 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:35.728567 25054 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:35.729044 24769 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:35.729401 22651 raft_consensus.cc:2468] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:35.714915 24849 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:35.727661 25074 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:35.730506 24877 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:35.731799 22658 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:35.698690 25045 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:35.713356 22653 raft_consensus.cc:2468] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:35.709472 22725 raft_consensus.cc:1240] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251026 14:59:35.675457 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:35.711928 25026 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:35.731606 24835 consensus_queue.cc:1048] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.007s
I20251026 14:59:35.739547 22656 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:35.740731 24527 raft_consensus.cc:3055] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:35.741146 24527 raft_consensus.cc:740] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:35.730379 24849 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.742077 24527 consensus_queue.cc:260] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.743417 24849 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:35.743772 22657 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:35.745265 22731 raft_consensus.cc:2468] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:35.745445 24851 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:35.751066 25037 raft_consensus.cc:3055] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:35.751575 25037 raft_consensus.cc:740] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:35.752396 25037 consensus_queue.cc:260] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.755370 25037 raft_consensus.cc:3060] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:35.755960 25041 consensus_queue.cc:1059] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 14:59:35.757434 24031 raft_consensus.cc:3060] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:35.757570 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:35.758495 22803 raft_consensus.cc:2410] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20251026 14:59:35.755522 22624 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:35.766057 22658 raft_consensus.cc:2468] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:35.712100 25114 raft_consensus.cc:697] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:35.774865 22700 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:35.776667 22700 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:35.777956 22624 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:35.779172 24332 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:35.777943 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 104306us
I20251026 14:59:35.779194 24769 raft_consensus.cc:2749] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20251026 14:59:35.758653 24921 consensus_queue.cc:1175] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "562bb1e1c8e445b8baecd29c3669658e" responder_term: 5 status { last_received { term: 5 index: 4 } last_committed_idx: 3 last_received_current_leader { term: 5 index: 4 } }
I20251026 14:59:35.782804 25114 consensus_queue.cc:237] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.785852 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:35.795120 22624 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:35.797374 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:35.798516 22804 raft_consensus.cc:2393] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 7: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:35.780488 25045 raft_consensus.cc:2804] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:35.757128 25041 consensus_queue.cc:1059] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20251026 14:59:35.781448 25124 raft_consensus.cc:3055] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Stepping down as leader of term 5
W20251026 14:59:35.802605 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:35.804049 22624 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:35.780932 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 49143.7us
I20251026 14:59:35.774935 24945 consensus_queue.cc:1048] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.028s
I20251026 14:59:35.816557 24849 raft_consensus.cc:2749] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:35.817195 25038 raft_consensus.cc:2804] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:35.798065 25095 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:35.805249 25124 raft_consensus.cc:740] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:35.830054 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:35.835136 25038 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:35.803857 25045 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:59:35.836704 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 51371us
I20251026 14:59:35.837383 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:35.838532 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:35.853675 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:35.854228 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:35.838913 24769 raft_consensus.cc:3055] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:35.855265 24769 raft_consensus.cc:740] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:35.856191 24853 consensus_queue.cc:1059] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:59:35.856101 24769 consensus_queue.cc:260] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.857321 24853 raft_consensus.cc:2749] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:35.858718 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:35.860430 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.860937 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:35.862052 22651 raft_consensus.cc:2410] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251026 14:59:35.862555 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.864135 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:35.864723 22654 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:35.792619 24332 raft_consensus.cc:2749] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:35.846050 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:35.869829 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:35.871580 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:35.872654 22801 raft_consensus.cc:2393] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 7: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:35.847223 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:35.848981 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.873875 22656 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:35.842554 25095 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:35.841482 22802 raft_consensus.cc:1240] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:35.850427 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.843111 24843 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:35.851475 25038 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:35.876665 22657 raft_consensus.cc:2468] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:35.876696 24843 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.879556 25045 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:35.857393 24769 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:35.840764 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:35.881852 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:35.862084 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:35.885834 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:35.886986 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.817080 25054 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.853346 24885 consensus_queue.cc:1059] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20251026 14:59:35.900825 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:35.955153 22658 raft_consensus.cc:2468] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:35.958237 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:35.959196 22807 raft_consensus.cc:2393] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:35.851806 25041 consensus_queue.cc:1059] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20251026 14:59:35.959234 22654 raft_consensus.cc:2468] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:35.973917 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:35.974256 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 197639us
I20251026 14:59:35.978494 22700 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:35.980098 22700 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:35.980177 24945 raft_consensus.cc:2764] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20251026 14:59:35.980934 24849 raft_consensus.cc:2804] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:35.981033 22700 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:35.981304 22650 raft_consensus.cc:2468] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:35.981434 24849 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:35.981937 24849 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:35.982049 24945 raft_consensus.cc:2804] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:35.984884 24031 raft_consensus.cc:3055] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:35.985745 24031 raft_consensus.cc:740] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:35.982600 22700 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:35.986125 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.986759 22658 raft_consensus.cc:2468] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:35.987273 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:35.988974 22653 raft_consensus.cc:2393] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 6: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:35.988080 24945 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:35.995441 24945 raft_consensus.cc:3060] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:35.995766 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:35.996543 22803 raft_consensus.cc:2393] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 4: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:35.996590 24843 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:35.997538 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:35.997794 24031 consensus_queue.cc:260] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:35.999058 24031 raft_consensus.cc:3060] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:36.001453 25152 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:36.001655 25041 raft_consensus.cc:3055] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:36.002023 25041 raft_consensus.cc:740] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:36.002074 25152 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.002902 25041 consensus_queue.cc:260] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.004280 25041 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:36.014878 22658 raft_consensus.cc:2468] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:36.015169 25026 raft_consensus.cc:2804] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251026 14:59:35.849182 25074 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.017853 25026 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:36.020604 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:36.020574 25026 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:36.022462 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:36.022835 25054 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:36.023154 22651 raft_consensus.cc:2376] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6. Re-sending same reply.
I20251026 14:59:35.969468 25121 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.024106 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:36.019485 25074 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:36.026264 25121 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:36.016417 22804 raft_consensus.cc:2468] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:36.020282 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:36.040169 22655 raft_consensus.cc:1275] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:36.019941 22700 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:36.045830 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:36.047331 22659 raft_consensus.cc:2393] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 7: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:36.047655 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:36.048481 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:36.004302 25152 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:36.051878 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:36.050259 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 246267us
I20251026 14:59:36.056555 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.055573 22624 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:36.058064 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:36.058590 22654 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:36.058329 22624 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:36.060546 22659 raft_consensus.cc:2368] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8.
I20251026 14:59:36.061666 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:36.067268 22659 raft_consensus.cc:1275] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:36.069226 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:36.019644 24945 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.070950 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:36.074347 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.062939 25152 raft_consensus.cc:2749] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:36.086158 22624 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:36.098094 25121 raft_consensus.cc:2749] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:36.053894 25026 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.100939 25026 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:36.109671 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.110098 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.111006 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:36.111658 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.112107 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.112900 22807 raft_consensus.cc:2393] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 6: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:36.113261 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:36.114421 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.114547 22805 raft_consensus.cc:2393] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 6: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:36.115401 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:36.115926 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.116698 22803 raft_consensus.cc:2393] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 6: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:36.125933 25038 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:36.128197 25038 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
W20251026 14:59:36.128533 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:36.129701 25045 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.131691 25045 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:36.088169 25037 raft_consensus.cc:2749] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:36.113575 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.137625 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:36.114717 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.141403 22808 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:36.086330 25158 raft_consensus.cc:2764] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20251026 14:59:36.153478 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:36.158703 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.158892 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.158723 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.163481 22801 raft_consensus.cc:2410] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251026 14:59:36.157604 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.164250 22651 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:36.165616 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:36.166795 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:36.167673 22655 raft_consensus.cc:2468] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:36.170310 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:36.168262 22806 raft_consensus.cc:1275] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:36.170982 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.171671 22808 raft_consensus.cc:2393] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:36.173101 22654 raft_consensus.cc:2393] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 8: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:36.173671 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:36.174721 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.176411 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.178579 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:36.183032 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:36.174633 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:36.184732 22655 raft_consensus.cc:2410] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251026 14:59:36.193006 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:59:36.193504 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 173634us
I20251026 14:59:36.193812 22651 raft_consensus.cc:1240] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:36.193984 22654 raft_consensus.cc:3060] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:36.194249 22700 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:36.195428 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:36.205785 22700 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:36.208114 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 110009us
I20251026 14:59:36.210729 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.212519 22655 raft_consensus.cc:1240] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:36.216239 22651 raft_consensus.cc:1240] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:36.209398 25158 raft_consensus.cc:2804] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:36.211148 25045 raft_consensus.cc:2804] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:36.219848 22700 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:36.216786 24945 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:36.221138 24945 raft_consensus.cc:2804] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:36.221571 24945 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:36.221976 24945 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:36.227470 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.230180 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.230883 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:36.230991 22806 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
I20251026 14:59:36.231618 22731 raft_consensus.cc:2393] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 7: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:36.231992 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.232084 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:36.232887 22799 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 9.
I20251026 14:59:36.233031 22727 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 3.
I20251026 14:59:36.237676 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.238551 22808 raft_consensus.cc:2393] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 6: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:36.239853 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:36.243681 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:36.244840 22806 raft_consensus.cc:2393] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 7: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:36.245628 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:36.245628 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:36.246291 22803 raft_consensus.cc:3055] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:36.246589 22727 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:36.246749 22803 raft_consensus.cc:740] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:36.249496 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:36.249163 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:36.251289 22803 consensus_queue.cc:260] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.251694 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:36.253492 22803 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:36.253758 22728 raft_consensus.cc:2410] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20251026 14:59:36.249769 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.256131 22805 raft_consensus.cc:2393] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 7: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:36.218221 25158 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:36.258045 22729 raft_consensus.cc:2410] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251026 14:59:36.258466 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:36.257438 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:36.267033 22734 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:36.260982 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:36.268172 22732 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 8
I20251026 14:59:36.276662 22654 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:36.229383 25045 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:36.310401 22654 raft_consensus.cc:1275] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:36.329629 22624 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:36.282881 25158 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:36.340308 22732 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:36.344642 22803 raft_consensus.cc:2468] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:36.349913 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:36.352741 22624 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:36.354915 22773 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:36.361418 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:36.364569 22624 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:36.313845 25167 log.cc:927] Time spent T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
I20251026 14:59:36.367019 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:36.365945 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:36.368659 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:36.313454 25045 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
W20251026 14:59:36.369822 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:36.369554 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:36.371003 22624 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:36.371455 22622 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:36.371421 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:36.378424 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:36.361662 25165 log.cc:927] Time spent T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.149s user 0.000s sys 0.001s
I20251026 14:59:36.384528 22730 raft_consensus.cc:2393] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 5: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:36.385401 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:36.386077 22731 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:36.387368 22726 raft_consensus.cc:2393] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 6: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:36.388548 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:36.390516 22774 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.390776 22727 raft_consensus.cc:1275] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:36.391923 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:36.393013 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:36.372829 25037 raft_consensus.cc:2749] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251026 14:59:36.396212 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:36.398197 22655 raft_consensus.cc:2468] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:36.398988 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:36.386386 22624 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:36.363624 25058 raft_consensus.cc:2749] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:36.369519 25114 raft_consensus.cc:2804] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:36.403656 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:36.403903 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.404450 22657 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
W20251026 14:59:36.405458 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 196232us
I20251026 14:59:36.407083 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:36.407740 22774 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.377682 25038 raft_consensus.cc:2749] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:36.387037 24843 raft_consensus.cc:2764] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20251026 14:59:36.390192 25041 raft_consensus.cc:2749] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:36.412004 22624 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:36.418443 22774 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:36.420746 22725 raft_consensus.cc:1217] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[5.3-9.5] Dedup: 5.3->[7.4-9.5]
I20251026 14:59:36.421300 22725 raft_consensus.cc:1240] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 9. Current term is 10. Ops: [5.3-9.5]
I20251026 14:59:36.413304 22731 raft_consensus.cc:2955] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 2 to 6, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:59:36.383924 25152 consensus_queue.cc:1059] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20251026 14:59:36.435384 25095 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:36.410374 25074 raft_consensus.cc:2804] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:36.428184 22655 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:36.428897 24740 raft_consensus.cc:2804] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251026 14:59:36.429629 22604 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica df3a0d1619ee48e8ab68ee0543a72a29 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:36.430080 22697 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.429869 24945 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:36.413887 24835 raft_consensus.cc:2749] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:36.425981 22734 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:36.437801 24877 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:36.449342 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:36.449981 22734 raft_consensus.cc:3060] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 5
I20251026 14:59:36.439323 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:36.421416 24873 raft_consensus.cc:2749] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:36.419829 25114 raft_consensus.cc:697] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:36.421671 25058 raft_consensus.cc:2749] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:36.456499 22774 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.454042 23370 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.457449 24877 raft_consensus.cc:2749] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:36.463250 24527 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:36.464058 25144 raft_consensus.cc:2749] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:36.464962 25095 raft_consensus.cc:2749] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251026 14:59:36.465318 25166 log.cc:927] Time spent T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.359s user 0.000s sys 0.003s
I20251026 14:59:36.438486 25074 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:36.467969 25074 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
W20251026 14:59:36.468551 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 52065.7us
I20251026 14:59:36.470713 24945 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:36.470928 22734 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:36.471679 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:36.472549 22656 raft_consensus.cc:2410] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20251026 14:59:36.477844 22734 raft_consensus.cc:1275] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:36.475557 22656 raft_consensus.cc:1275] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20251026 14:59:36.478827 22656 pending_rounds.cc:85] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29: Aborting all ops after (but not including) 3
I20251026 14:59:36.479229 22656 pending_rounds.cc:107] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20251026 14:59:36.479811 22656 raft_consensus.cc:2889] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:36.468796 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:36.485953 22652 raft_consensus.cc:2368] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8.
I20251026 14:59:36.442456 24740 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:36.433254 24849 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.502609 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:36.499241 22774 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:36.495599 22655 raft_consensus.cc:1275] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:36.506916 22697 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.506378 22657 raft_consensus.cc:2955] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Committing config change with OpId 6.6: config changed from index 2 to 6, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:59:36.508500 22653 raft_consensus.cc:1275] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20251026 14:59:36.497659 24740 raft_consensus.cc:3060] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:36.514147 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:36.518016 22725 raft_consensus.cc:3060] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:36.518824 22733 raft_consensus.cc:1240] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20251026 14:59:36.514916 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:36.522053 22726 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:36.522123 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:36.515823 22729 raft_consensus.cc:1240] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:36.528808 25037 raft_consensus.cc:3055] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:36.529373 25037 raft_consensus.cc:740] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:36.530184 25037 consensus_queue.cc:260] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.535570 22726 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:36.535776 25037 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:36.536865 22726 raft_consensus.cc:1240] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:36.539003 25074 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.510041 24945 raft_consensus.cc:2749] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:36.542371 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:36.546303 22697 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.547215 22726 raft_consensus.cc:1275] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:36.547576 22697 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.548393 22726 raft_consensus.cc:1240] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:36.548671 22697 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.549891 25178 raft_consensus.cc:2804] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:36.550503 25178 raft_consensus.cc:697] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:36.551414 25178 consensus_queue.cc:237] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.557260 22554 catalog_manager.cc:5649] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "248493ec312e4398b828c6cb00e5c4b1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 14:59:36.559022 22554 catalog_manager.cc:5649] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: term changed from 4 to 5, leader changed from 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 5 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:59:36.559942 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:36.560535 22803 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:36.561656 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:36.561949 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.562847 22807 raft_consensus.cc:2410] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20251026 14:59:36.565516 22654 raft_consensus.cc:1240] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:36.565680 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.567862 22730 raft_consensus.cc:1275] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20251026 14:59:36.569983 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:36.573284 22697 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.578827 22725 raft_consensus.cc:2410] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251026 14:59:36.560489 22554 catalog_manager.cc:5649] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: NON_VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: true } } }
I20251026 14:59:36.598115 24877 raft_consensus.cc:2749] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:36.607475 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:36.608287 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:36.608951 22652 raft_consensus.cc:3060] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:36.609258 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:36.564815 25158 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.575279 24873 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:36.611168 22604 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 4129daa4fb70408193f07b8e6d25effe, TS 0d684445287b450a8f738b6ad4b59821 -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:36.624300 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 51156us
I20251026 14:59:36.599776 24945 raft_consensus.cc:2749] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251026 14:59:36.638432 22604 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 4129daa4fb70408193f07b8e6d25effe, TS 0d684445287b450a8f738b6ad4b59821 -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.639187 22697 leader_election.cc:304] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:36.620872 22734 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:36.626827 22658 raft_consensus.cc:2468] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:36.594736 25026 raft_consensus.cc:2749] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:36.504381 25124 consensus_queue.cc:260] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:36.663084 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 163930us
I20251026 14:59:36.665763 24031 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:36.666468 24031 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.669144 22733 raft_consensus.cc:1275] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:36.670228 24031 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:36.664857 22774 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.677999 22624 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:36.637759 24877 raft_consensus.cc:3060] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:36.686188 22624 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:36.693454 22624 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:36.695145 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:36.700984 24031 raft_consensus.cc:2804] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:36.701814 24031 raft_consensus.cc:697] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:36.696929 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:36.661098 22697 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.658023 22803 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:36.704319 22697 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.713006 22732 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:36.662772 25031 raft_consensus.cc:2749] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:36.716500 22733 raft_consensus.cc:1240] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:36.718634 24031 consensus_queue.cc:237] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.720714 22650 raft_consensus.cc:1240] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:36.699280 25041 raft_consensus.cc:2804] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:36.645438 24873 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.618674 25114 consensus_queue.cc:237] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.706769 24748 raft_consensus.cc:2749] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251026 14:59:36.706921 25098 raft_consensus.cc:2749] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251026 14:59:36.705781 25026 raft_consensus.cc:2749] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:36.704949 25074 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:36.732149 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 67390.1us
I20251026 14:59:36.681289 25144 consensus_queue.cc:1048] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.044s
I20251026 14:59:36.736333 25144 consensus_queue.cc:1243] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer df3a0d1619ee48e8ab68ee0543a72a29 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251026 14:59:36.737673 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:36.739111 22726 raft_consensus.cc:1275] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20251026 14:59:36.739984 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 57816.8us
I20251026 14:59:36.742208 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.748709 22622 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:36.752161 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:36.750842 22773 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.754601 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.753649 22622 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:36.728224 24945 raft_consensus.cc:2749] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:36.755623 22774 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:36.758715 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:36.766671 22774 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.715920 25045 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.768266 25144 raft_consensus.cc:2749] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:36.738590 25041 raft_consensus.cc:697] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:36.757551 25152 raft_consensus.cc:2749] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:36.769420 25098 raft_consensus.cc:2804] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:36.779521 25158 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:36.751066 24849 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:36.799553 25098 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:36.785352 25179 log.cc:927] Time spent T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.069s user 0.000s sys 0.001s
W20251026 14:59:36.799953 25176 log.cc:927] Time spent T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20251026 14:59:36.800065 25098 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:36.791303 22652 raft_consensus.cc:2468] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:36.791740 22774 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:36.802098 24877 raft_consensus.cc:2749] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:36.803396 22774 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:36.803992 25045 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:36.804379 25185 raft_consensus.cc:2804] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election won for term 7
W20251026 14:59:36.794974 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 50525.9us
I20251026 14:59:36.806334 22773 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.766134 24527 raft_consensus.cc:2749] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251026 14:59:36.795286 24835 raft_consensus.cc:2749] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251026 14:59:36.816713 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 110985us
W20251026 14:59:36.819566 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:36.826157 22700 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:36.833454 22700 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:36.814638 25144 raft_consensus.cc:2804] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:36.844370 22700 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:36.847908 22773 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.850068 22700 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:36.852870 22773 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:36.854224 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 149993us
I20251026 14:59:36.795910 25186 consensus_queue.cc:1048] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20251026 14:59:36.843250 25045 raft_consensus.cc:2764] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20251026 14:59:36.843986 24843 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:36.855135 22697 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.856825 22700 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:36.848172 24877 raft_consensus.cc:2804] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:36.855090 25185 raft_consensus.cc:697] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:36.850677 24849 raft_consensus.cc:2804] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election won for term 6
W20251026 14:59:36.869261 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 65785.3us
I20251026 14:59:36.869153 22658 raft_consensus.cc:1240] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:36.852887 24835 raft_consensus.cc:2804] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:36.873216 22774 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.852573 25144 raft_consensus.cc:697] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:36.875110 22774 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.872146 22697 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.875823 22700 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:36.842041 25098 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.858379 24873 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:36.870414 24877 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:36.859934 25114 raft_consensus.cc:2749] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:36.866442 25158 raft_consensus.cc:2804] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:36.859983 25194 consensus_queue.cc:1048] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20251026 14:59:36.886363 22697 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:36.885680 25195 raft_consensus.cc:2749] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:36.888105 25031 raft_consensus.cc:2749] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:36.886116 24877 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:36.880460 25159 consensus_queue.cc:1059] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.009s
I20251026 14:59:36.890857 25098 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:36.879158 25192 raft_consensus.cc:2804] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:36.914930 25192 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:36.915381 25192 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:36.871999 24945 raft_consensus.cc:2749] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251026 14:59:36.906608 25158 raft_consensus.cc:697] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:36.887732 25045 raft_consensus.cc:2804] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:36.917888 25045 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:36.918309 25045 raft_consensus.cc:3060] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:36.918368 25158 consensus_queue.cc:237] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.921523 25193 raft_consensus.cc:2749] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:36.921985 25159 raft_consensus.cc:3055] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:36.922389 25159 raft_consensus.cc:740] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:36.923274 25159 consensus_queue.cc:260] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:36.924449 25159 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:36.882931 24849 raft_consensus.cc:697] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:36.876436 24835 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:36.895476 24873 raft_consensus.cc:2749] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:36.894657 25185 consensus_queue.cc:237] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.895691 25144 consensus_queue.cc:237] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.880560 22700 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:36.904409 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 51627.9us
I20251026 14:59:36.869851 24843 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.906605 22697 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.858081 25041 consensus_queue.cc:237] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.909533 25026 raft_consensus.cc:2804] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election won for term 6
W20251026 14:59:36.917033 25183 log.cc:927] Time spent T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.212s user 0.002s sys 0.000s
I20251026 14:59:36.893216 24740 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:36.929885 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 49411.2us
I20251026 14:59:36.815575 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:59:36.929181 24835 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:36.934415 22697 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.932957 24945 raft_consensus.cc:2804] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:36.938839 22700 leader_election.cc:400] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8.
I20251026 14:59:36.934700 25171 raft_consensus.cc:2749] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:36.939194 22700 leader_election.cc:403] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20251026 14:59:36.938869 22773 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:36.909305 25114 consensus_queue.cc:1059] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20251026 14:59:36.941416 22774 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:36.947012 25031 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:36.948252 25171 raft_consensus.cc:2749] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8.
I20251026 14:59:36.935757 25026 raft_consensus.cc:697] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:36.949685 25114 raft_consensus.cc:3055] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:36.945969 25194 raft_consensus.cc:2749] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:36.950048 22700 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:36.950431 25026 consensus_queue.cc:237] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.950062 25171 consensus_queue.cc:1059] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20251026 14:59:36.933187 24849 consensus_queue.cc:237] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.950083 25114 raft_consensus.cc:740] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:36.952991 25114 consensus_queue.cc:260] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.954385 25114 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:36.948328 24843 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:36.959746 22774 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:36.961591 25194 raft_consensus.cc:2804] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:36.964196 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:36.964327 25194 raft_consensus.cc:697] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:36.965055 22802 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
I20251026 14:59:36.965649 25194 consensus_queue.cc:237] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.966557 22622 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:36.961503 25195 consensus_queue.cc:1048] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20251026 14:59:36.968904 24843 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251026 14:59:36.971747 24843 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:36.974251 24843 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:36.977375 25195 raft_consensus.cc:2804] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:36.978220 25195 raft_consensus.cc:697] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:36.979475 25195 consensus_queue.cc:237] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:36.982383 24740 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:36.983496 25124 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:36.990357 22774 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:37.005455 22774 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:37.014991 24945 raft_consensus.cc:697] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:37.015909 25198 raft_consensus.cc:2804] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:37.016491 25198 raft_consensus.cc:697] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:37.016598 24945 consensus_queue.cc:237] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:37.017524 25198 consensus_queue.cc:237] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:37.018368 25171 raft_consensus.cc:3055] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:36.961302 25197 consensus_queue.cc:1059] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.074s
I20251026 14:59:37.025620 22774 leader_election.cc:400] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8.
I20251026 14:59:37.029161 22774 leader_election.cc:403] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20251026 14:59:37.018873 24873 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:37.019101 25194 raft_consensus.cc:2804] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election won for term 8
W20251026 14:59:37.035218 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 84241.8us
I20251026 14:59:37.036811 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:37.037801 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:37.038975 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:37.039443 25031 raft_consensus.cc:2749] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:37.046964 22624 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:37.047870 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:37.048816 22652 raft_consensus.cc:2468] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:37.050227 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:37.051069 22656 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:37.051862 22728 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:37.044081 24877 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:37.027767 25171 raft_consensus.cc:740] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:37.057870 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:37.060704 25152 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:37.061291 25152 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:37.035989 25197 consensus_queue.cc:1243] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer 562bb1e1c8e445b8baecd29c3669658e log is divergent from this leader: its last log entry 8.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20251026 14:59:37.069615 22729 raft_consensus.cc:1240] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 5. Current term is 6. Ops: []
W20251026 14:59:37.071446 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 46041.5us
I20251026 14:59:37.072104 22700 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:37.069808 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:37.076366 22624 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:37.077876 24031 raft_consensus.cc:2804] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:37.078253 24031 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:37.078509 24031 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:37.082418 25152 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:37.084249 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:37.053498 24748 raft_consensus.cc:2749] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251026 14:59:37.084936 22802 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:37.054979 25178 raft_consensus.cc:2749] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8.
I20251026 14:59:37.053613 25194 raft_consensus.cc:697] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 8 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:37.087476 22622 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:37.029768 25196 consensus_queue.cc:1048] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
W20251026 14:59:37.089038 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 147146us
I20251026 14:59:37.107369 22658 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:37.084874 25031 raft_consensus.cc:2804] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:37.099604 22653 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:37.110672 22700 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:37.110675 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:37.124683 22728 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:37.108866 25058 raft_consensus.cc:2804] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:37.128319 25058 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:37.128695 25058 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:37.129112 24877 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:37.123562 25031 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:37.123809 25195 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:37.129747 25031 raft_consensus.cc:3060] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:37.130105 25195 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:37.132269 23369 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:37.132215 25195 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:37.135106 22700 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:37.136332 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:37.141763 24031 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:37.142941 22651 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:37.085230 25098 consensus_queue.cc:1048] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20251026 14:59:37.086563 25197 consensus_queue.cc:1048] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.049s
I20251026 14:59:37.148013 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:37.151980 22728 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:37.157824 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:37.158707 25194 consensus_queue.cc:237] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:37.161738 25158 raft_consensus.cc:2804] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:37.162274 25158 raft_consensus.cc:697] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:37.163146 25158 consensus_queue.cc:237] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:37.142253 25199 raft_consensus.cc:2804] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:37.149746 24031 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:37.177758 22656 raft_consensus.cc:2468] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:37.088613 25193 consensus_queue.cc:1059] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20251026 14:59:37.166630 25199 raft_consensus.cc:697] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:37.183614 22700 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:37.190006 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 101941us
I20251026 14:59:37.190812 22774 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:37.160974 25192 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:37.126672 25171 consensus_queue.cc:260] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:37.179683 25186 consensus_queue.cc:1048] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.026s
I20251026 14:59:37.219231 22652 raft_consensus.cc:1240] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: [2.2-6.4]
I20251026 14:59:37.197933 24877 raft_consensus.cc:2804] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:37.224545 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:37.226085 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:37.233247 22656 raft_consensus.cc:1240] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:37.239467 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:37.240500 22656 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
W20251026 14:59:37.205782 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 271380us
I20251026 14:59:37.210832 22651 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:37.245858 22650 raft_consensus.cc:1240] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20251026 14:59:37.213953 22653 raft_consensus.cc:1275] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20251026 14:59:37.215560 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:37.217934 22658 raft_consensus.cc:1275] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:37.251986 22697 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:37.249651 22700 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:37.254900 22728 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
W20251026 14:59:37.259775 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:37.266430 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 163737us
I20251026 14:59:37.266506 22657 raft_consensus.cc:2468] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:37.268405 22773 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:37.247119 25195 raft_consensus.cc:2804] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:37.221463 24835 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:37.286914 22658 raft_consensus.cc:1240] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:37.231721 24843 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:37.257467 25171 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:37.295399 22802 raft_consensus.cc:1275] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20251026 14:59:37.295979 22802 pending_rounds.cc:85] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821: Aborting all ops after (but not including) 5
I20251026 14:59:37.296274 22802 pending_rounds.cc:107] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821: Aborting uncommitted NO_OP operation due to leader change: 7.6
I20251026 14:59:37.296557 22802 raft_consensus.cc:2889] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:37.290318 25026 raft_consensus.cc:2749] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:37.307128 25058 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:37.338799 25058 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:37.307552 25210 raft_consensus.cc:3055] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:37.341053 25210 raft_consensus.cc:740] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:37.341986 25210 consensus_queue.cc:260] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:37.343420 25210 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:37.307818 25211 consensus_queue.cc:1059] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:59:37.348191 25045 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:37.309918 24835 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:37.313217 25212 raft_consensus.cc:2804] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:37.301740 25214 consensus_queue.cc:1059] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20251026 14:59:37.298336 25195 raft_consensus.cc:697] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:37.318079 25209 raft_consensus.cc:2749] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20251026 14:59:37.320681 25205 log.cc:927] Time spent T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.283s user 0.002s sys 0.000s
I20251026 14:59:37.350698 24843 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:37.290351 25199 consensus_queue.cc:237] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:37.357316 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 105387us
I20251026 14:59:37.400450 22697 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:37.361663 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 114535us
I20251026 14:59:37.369784 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:37.404665 22808 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Advancing to term 10
W20251026 14:59:37.407205 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:37.407632 22657 raft_consensus.cc:1275] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:37.378394 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:37.409755 22800 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:37.424077 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:37.426576 22655 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:37.428555 25214 consensus_queue.cc:1243] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer df3a0d1619ee48e8ab68ee0543a72a29 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251026 14:59:37.429643 25195 consensus_queue.cc:237] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:37.429188 25215 raft_consensus.cc:3055] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:37.430658 25215 raft_consensus.cc:740] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:37.431520 22729 raft_consensus.cc:1275] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251026 14:59:37.431475 25215 consensus_queue.cc:260] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:37.377753 25212 raft_consensus.cc:697] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:37.432763 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:37.433866 25212 consensus_queue.cc:237] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:37.364686 25193 raft_consensus.cc:3055] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:37.437234 25193 raft_consensus.cc:740] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:37.433413 22732 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 9
I20251026 14:59:37.438081 25193 consensus_queue.cc:260] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:37.439100 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:37.439344 25192 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:37.434175 22624 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:37.442250 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:37.445302 22658 raft_consensus.cc:1275] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251026 14:59:37.447428 24031 consensus_queue.cc:1048] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20251026 14:59:37.450172 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 180491us
I20251026 14:59:37.451263 22774 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:37.433162 25215 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:37.449345 25152 raft_consensus.cc:2804] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:37.506660 22808 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 10.
I20251026 14:59:37.454627 25192 consensus_queue.cc:1048] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.104s
I20251026 14:59:37.449383 25200 raft_consensus.cc:2749] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251026 14:59:37.511508 23369 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:37.642382 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:37.659513 22732 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
I20251026 14:59:37.660027 22622 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:37.660597 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 260251us
I20251026 14:59:37.580286 25216 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:37.671998 22774 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:37.688750 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:37.586922 25219 consensus_queue.cc:1048] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.519s
I20251026 14:59:38.047901 25226 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:37.756574 25186 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
W20251026 14:59:37.583343 25124 consensus_meta.cc:287] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.599s user 0.012s sys 0.000s
I20251026 14:59:37.697604 25214 consensus_queue.cc:1048] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.381s
W20251026 14:59:38.045476 25195 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.006s sys 0.013s
I20251026 14:59:37.535809 22655 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
W20251026 14:59:37.611503 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 343195us
I20251026 14:59:37.757113 25209 consensus_queue.cc:1048] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.324s
W20251026 14:59:37.763248 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 319626us
I20251026 14:59:37.618479 25045 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:37.457607 24877 raft_consensus.cc:697] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
W20251026 14:59:37.798839 22555 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38878 (request call id 163) took 23900 ms (client timeout 15000 ms). Trace:
W20251026 14:59:38.074803 22800 consensus_meta.cc:287] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.631s user 0.016s sys 0.008s
I20251026 14:59:38.100651 22800 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
I20251026 14:59:37.470834 24835 raft_consensus.cc:2804] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:38.077836 22652 raft_consensus.cc:3060] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
W20251026 14:59:38.090487 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 418585us
I20251026 14:59:38.086898 22659 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
W20251026 14:59:38.086433 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 426497us
I20251026 14:59:38.118824 22808 raft_consensus.cc:1275] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251026 14:59:38.082068 22697 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:37.579368 24873 consensus_meta.cc:287] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.542s user 0.006s sys 0.004s
W20251026 14:59:37.771445 25208 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.621s user 0.008s sys 0.019s
W20251026 14:59:37.639688 25031 consensus_meta.cc:287] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.509s user 0.011s sys 0.002s
W20251026 14:59:38.127365 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 45353.5us
I20251026 14:59:38.134008 22697 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:38.079581 25195 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.587s user 0.007s sys 0.014s
W20251026 14:59:38.049599 25199 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.003s sys 0.011s
I20251026 14:59:38.085397 25121 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:38.135377 22729 raft_consensus.cc:1275] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251026 14:59:38.108566 24835 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:38.183751 22771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 102230us
I20251026 14:59:38.121785 24843 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:38.099566 25124 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:59:38.130174 25208 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.976s user 0.009s sys 0.032s
I20251026 14:59:38.106370 25144 raft_consensus.cc:3055] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 9 LEADER]: Stepping down as leader of term 9
W20251026 14:59:38.184330 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 103203us
W20251026 14:59:38.219811 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 85871.5us
W20251026 14:59:38.220430 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:38.137924 25199 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.696s user 0.004s sys 0.011s
W20251026 14:59:38.245505 22604 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 5b0536aab3b541a2a846d187e111c09c, TS 0d684445287b450a8f738b6ad4b59821 -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:38.227747 23369 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.509s (SENT) [suppressed 21 similar messages]
I20251026 14:59:38.075917 25216 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.137202 25200 raft_consensus.cc:2749] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:38.077235 25226 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:38.103448 22555 rpcz_store.cc:269] 1026 14:59:13.889794 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:13.902535 (+ 12741us) service_pool.cc:225] Handling call
1026 14:59:37.789605 (+23887070us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 14:59:35.462902 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 14:59:35.463780 (+ 878us) write_op.cc:432] Acquiring schema lock in shared mode
1026 14:59:35.463839 (+ 59us) write_op.cc:435] Acquired schema lock
1026 14:59:35.463873 (+ 34us) tablet.cc:660] Decoding operations
1026 14:59:35.465248 (+ 1375us) write_op.cc:620] Acquiring the partition lock for write op
1026 14:59:35.465369 (+ 121us) write_op.cc:641] Partition lock acquired for write op
1026 14:59:35.465399 (+ 30us) tablet.cc:683] Acquiring locks for 9 operations
1026 14:59:35.466541 (+ 1142us) tablet.cc:699] Row locks acquired
1026 14:59:35.466582 (+ 41us) write_op.cc:260] PREPARE: finished
1026 14:59:35.466765 (+ 183us) write_op.cc:270] Start()
1026 14:59:35.466944 (+ 179us) write_op.cc:276] Timestamp: P: 1761490775466699 usec, L: 0
1026 14:59:35.466964 (+ 20us) op_driver.cc:348] REPLICATION: starting
1026 14:59:35.467767 (+ 803us) log.cc:844] Serialized 3564 byte log entry
1026 14:59:35.775491 (+307724us) op_driver.cc:464] REPLICATION: finished
1026 14:59:35.821400 (+ 45909us) write_op.cc:301] APPLY: starting
1026 14:59:35.821499 (+ 99us) tablet.cc:1365] starting BulkCheckPresence
1026 14:59:35.821824 (+ 325us) tablet.cc:1368] finished BulkCheckPresence
1026 14:59:35.821981 (+ 157us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 14:59:35.835816 (+ 13835us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 14:59:35.836210 (+ 394us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
1026 14:59:35.836284 (+ 74us) write_op.cc:312] APPLY: finished
1026 14:59:35.836611 (+ 327us) log.cc:844] Serialized 70 byte log entry
1026 14:59:35.836820 (+ 209us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:35.836961 (+ 141us) write_op.cc:454] Released schema lock
1026 14:59:35.837029 (+ 68us) write_op.cc:341] FINISH: Updating metrics
1026 14:59:35.939151 (+102122us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:35.939206 (+ 55us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1097500,"reported_tablets":37,"child_traces":[["op",{"apply.queue_time_us":45562,"apply.run_cpu_time_us":4007,"apply.run_wall_time_us":117654,"num_ops":9,"prepare.queue_time_us":268218,"prepare.run_cpu_time_us":16490,"prepare.run_wall_time_us":186318,"raft.queue_time_us":85036,"raft.run_cpu_time_us":10267,"raft.run_wall_time_us":19029,"replication_time_us":308368,"thread_start_us":467454,"threads_started":4,"wal-append.queue_time_us":219707,"wal-append.run_cpu_time_us":12325,"wal-append.run_wall_time_us":1153934}]]}
I20251026 14:59:38.242882 25045 raft_consensus.cc:3060] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:38.133256 25209 consensus_queue.cc:1059] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.115s
I20251026 14:59:38.242074 25233 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:38.242259 24835 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
W20251026 14:59:38.265497 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 139446us
I20251026 14:59:38.132869 25031 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.077450 25186 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:38.128908 24873 raft_consensus.cc:2749] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:38.126550 25197 raft_consensus.cc:2804] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:38.270171 22652 raft_consensus.cc:2468] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:38.243669 25231 consensus_queue.cc:1059] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.079s
W20251026 14:59:38.296389 22604 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 5b0536aab3b541a2a846d187e111c09c, TS 0d684445287b450a8f738b6ad4b59821 -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.239629 25121 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:38.326659 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 199464us
I20251026 14:59:38.332307 22774 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:38.274467 25144 raft_consensus.cc:740] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
W20251026 14:59:38.337738 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 74212.1us
W20251026 14:59:38.339589 25225 log.cc:927] Time spent T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
W20251026 14:59:38.340420 25235 log.cc:927] Time spent T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20251026 14:59:38.274470 24843 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.275679 25124 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.331068 25239 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Leader election won for term 10
W20251026 14:59:38.371083 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 52688.4us
I20251026 14:59:38.327724 24740 consensus_queue.cc:1059] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20251026 14:59:38.383562 22700 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:38.315574 24877 consensus_queue.cc:237] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.443265 24031 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:59:38.443490 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 111274us
I20251026 14:59:38.443928 24031 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:38.447271 22774 connection.cc:232] RPC call timeout handler was delayed by -1.0809972889999244s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20390000000000003s for a response
I20251026 14:59:38.449257 25199 consensus_queue.cc:1048] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
W20251026 14:59:38.483908 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:38.491510 24031 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:38.334049 25233 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.372848 24873 consensus_queue.cc:1048] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.072s
I20251026 14:59:38.426532 25246 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:38.495926 25231 raft_consensus.cc:3055] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:38.446151 25237 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:38.446290 25239 raft_consensus.cc:697] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:38.572382 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1484) took 1355 ms. Trace:
I20251026 14:59:38.494807 25199 consensus_queue.cc:1243] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer 0d684445287b450a8f738b6ad4b59821 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20251026 14:59:38.584379 22651 rpcz_store.cc:276] 1026 14:59:37.217126 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:37.244487 (+ 27361us) service_pool.cc:225] Handling call
1026 14:59:37.244922 (+ 435us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:37.255254 (+ 10332us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:37.255298 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:37.275063 (+ 19765us) log.cc:844] Serialized 26 byte log entry
1026 14:59:37.449799 (+174736us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:37.449856 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:37.449951 (+ 95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:38.572091 (+1122140us) raft_consensus.cc:1685] finished
1026 14:59:38.572122 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:38.572298 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3625,"raft.queue_time_us":4069,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":20999,"thread_start_us":149012,"threads_started":1,"wal-append.queue_time_us":780904}
I20251026 14:59:38.585851 25237 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.554370 25193 raft_consensus.cc:3060] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:38.602492 25248 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:38.603120 25248 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.547991 25251 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:38.602692 25207 log.cc:927] Time spent T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 1.444s user 0.010s sys 0.027s
I20251026 14:59:38.609066 25251 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.611357 25237 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:38.446988 25144 consensus_queue.cc:260] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.552457 25195 raft_consensus.cc:3055] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:38.549840 25196 raft_consensus.cc:2804] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:38.612274 25195 raft_consensus.cc:740] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:38.608330 25257 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:38.612558 25196 raft_consensus.cc:697] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 8 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:38.612993 25257 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.548230 25256 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:38.613266 25195 consensus_queue.cc:260] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.613680 25196 consensus_queue.cc:237] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.614087 25256 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.614542 25195 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:38.615330 25257 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:38.611364 25251 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:38.615890 25257 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:38.616300 25251 consensus_queue.cc:1048] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251026 14:59:38.616379 25257 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.554996 25124 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:38.613694 25031 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:38.584054 25202 log.cc:927] Time spent T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 1.421s user 0.025s sys 0.017s
I20251026 14:59:38.612376 25144 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:38.553530 25231 raft_consensus.cc:740] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
W20251026 14:59:38.620393 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:38.616163 25256 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:38.597443 25197 raft_consensus.cc:697] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
W20251026 14:59:38.638795 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:38.545106 25249 consensus_queue.cc:1048] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.171s
I20251026 14:59:38.491876 24740 consensus_queue.cc:1048] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.129s
I20251026 14:59:38.621340 25248 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:38.669618 22651 raft_consensus.cc:1275] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:38.613008 24835 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.576956 25246 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.656776 25242 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
W20251026 14:59:38.684648 22604 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6d267e50950c4ef2b95c745e7103cafb, TS df3a0d1619ee48e8ab68ee0543a72a29 -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:38.723116 22604 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 6d267e50950c4ef2b95c745e7103cafb, TS df3a0d1619ee48e8ab68ee0543a72a29 -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.665650 25209 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:38.669173 25243 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:38.659701 25226 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:38.767835 25254 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:38.768242 25250 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:38.767724 25260 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:38.768790 25250 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.768657 25254 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.770452 25246 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:38.770869 25250 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:38.768865 25260 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:38.771616 25224 log.cc:927] Time spent T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.363s user 0.001s sys 0.000s
I20251026 14:59:38.673561 25231 consensus_queue.cc:260] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:38.691568 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 319534us
W20251026 14:59:38.673166 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 289689us
I20251026 14:59:38.671779 25252 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
W20251026 14:59:38.812386 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 454032us
I20251026 14:59:38.836133 22622 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:38.803354 22697 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:38.838987 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:38.665246 25253 raft_consensus.cc:2804] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:38.846989 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:38.717720 25255 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:38.737156 25215 consensus_meta.cc:287] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 1.209s user 0.010s sys 0.004s
I20251026 14:59:38.689172 25258 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:38.737598 25234 log.cc:927] Time spent T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.355s user 0.001s sys 0.000s
I20251026 14:59:38.689769 25186 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:59:38.853863 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:38.854827 22773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.500s user 0.013s sys 0.009s
W20251026 14:59:38.870829 22773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.516s user 0.013s sys 0.009s
I20251026 14:59:38.675107 25121 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:38.882206 25121 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:38.882702 25121 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.757169 25239 consensus_queue.cc:237] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.884717 25121 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:38.870779 25258 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.810959 25233 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:38.810808 25243 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.835116 25250 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:38.861958 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:38.684268 25248 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:38.727104 24873 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:38.866776 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:38.729578 25216 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:38.871295 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 538970us
I20251026 14:59:38.810137 25249 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:38.870895 22802 raft_consensus.cc:2468] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:38.879591 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:38.916687 22807 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:38.879876 25261 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:38.880352 24835 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:38.880729 25247 log.cc:927] Time spent T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.222s user 0.001s sys 0.000s
I20251026 14:59:38.929240 25267 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:38.795882 25045 raft_consensus.cc:2749] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:38.929841 25267 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:38.930148 25279 consensus_queue.cc:1048] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251026 14:59:38.773886 25260 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:38.893870 23369 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.016s sys 0.036s
W20251026 14:59:38.931448 23369 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.604s user 0.016s sys 0.038s
W20251026 14:59:38.885387 22604 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 143c213a935041868ad3b0d9584ad863, TS 562bb1e1c8e445b8baecd29c3669658e -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.887154 25041 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:38.900061 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:38.726960 25259 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:38.830070 25193 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:38.831815 25256 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:38.847194 25246 raft_consensus.cc:2804] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader election won for term 9
I20251026 14:59:38.905383 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:38.901437 22806 raft_consensus.cc:2468] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:38.915494 22657 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:38.864748 25231 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:38.920264 22773 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:38.888482 25215 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:38.925653 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:39.006767 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 86602.6us
I20251026 14:59:39.007524 22773 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:38.930855 25026 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:38.887805 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:38.886574 25258 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:38.879626 25196 raft_consensus.cc:493] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:39.000821 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:38.810685 25209 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.901002 25240 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:38.990535 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:38.901243 24527 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:39.009263 22799 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
W20251026 14:59:39.005155 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 201871us
I20251026 14:59:39.035163 22697 leader_election.cc:304] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:39.037871 22697 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:39.039498 22657 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:38.737438 25242 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:38.931706 25279 raft_consensus.cc:2749] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:38.996564 25257 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:38.885234 25121 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:38.718022 25197 consensus_queue.cc:237] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.932072 25267 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:38.931939 25031 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:39.012542 22604 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 143c213a935041868ad3b0d9584ad863, TS 562bb1e1c8e445b8baecd29c3669658e -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.890959 25254 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.045755 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.040062 24527 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:39.046900 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 246608us
I20251026 14:59:39.043825 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:39.018407 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:38.918563 25248 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.049652 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:38.860438 25252 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:39.009070 25253 raft_consensus.cc:697] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:39.043049 22774 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:39.039378 22697 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:39.013921 25246 raft_consensus.cc:697] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:38.918906 25250 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.013854 25026 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:38.933463 25249 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.067337 25257 consensus_queue.cc:1048] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
W20251026 14:59:39.082633 25228 log.cc:927] Time spent T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.719s user 0.003s sys 0.002s
I20251026 14:59:39.083504 25249 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.084117 25273 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:38.933657 25261 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:39.085566 22653 raft_consensus.cc:1275] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:39.031812 25186 raft_consensus.cc:2804] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:38.871687 25255 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.063539 22655 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:39.068608 25252 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:39.009804 25193 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:39.057049 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:39.010879 25041 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:39.094240 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 47999.5us
I20251026 14:59:39.074740 22697 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:39.042215 25275 raft_consensus.cc:2749] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:39.010620 25256 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:38.928681 24873 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:39.078611 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:39.098404 25209 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:39.098879 25256 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:39.099538 25248 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.078491 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:39.080070 25264 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:39.100436 25264 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.101080 25242 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:39.101490 25026 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.061393 25031 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.102485 25264 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:39.103214 22804 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 10.
I20251026 14:59:39.080662 25195 raft_consensus.cc:2804] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:39.061882 25258 raft_consensus.cc:2764] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20251026 14:59:39.081200 25243 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.081110 22800 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:39.041438 25240 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.106562 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:39.110360 22658 raft_consensus.cc:1275] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20251026 14:59:39.110633 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 277746us
I20251026 14:59:38.984181 25233 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:39.087137 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:39.084661 25273 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.121486 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.122354 22728 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
W20251026 14:59:39.088546 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 72820.8us
I20251026 14:59:39.124262 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.125237 22728 raft_consensus.cc:2393] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 8: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
W20251026 14:59:38.931749 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 704003us
I20251026 14:59:39.091420 22656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1483) took 1874 ms. Trace:
I20251026 14:59:39.129688 22656 rpcz_store.cc:276] 1026 14:59:37.216887 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:37.241889 (+ 25002us) service_pool.cc:225] Handling call
1026 14:59:37.242356 (+ 467us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:37.242991 (+ 635us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:37.243033 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:37.252677 (+ 9644us) log.cc:844] Serialized 26 byte log entry
1026 14:59:37.490737 (+238060us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:37.490805 (+ 68us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:37.490916 (+ 111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:39.081948 (+1591032us) spinlock_profiling.cc:243] Waited 2.09 ms on lock 0x7b440002304c. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8cdfc0905 00007fd8cdfc1322 00007fd8d0d0a231 00007fd8d0d05b5a 00007fd8d0cf991e 00007fd8d0cf8c66 00007fd8d0cf8888 00007fd8d0d022c1 00007fd8d0d02279 00007fd8d0d02209
1026 14:59:39.085463 (+ 3515us) raft_consensus.cc:1685] finished
1026 14:59:39.085487 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:39.091350 (+ 5863us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3879,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":3791,"spinlock_wait_cycles":4813568,"thread_start_us":199280,"threads_started":1,"wal-append.queue_time_us":988819}
I20251026 14:59:39.071913 25265 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:39.013362 25215 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:39.140074 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 65456.5us
I20251026 14:59:39.106714 25031 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.147472 25285 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:39.147699 25215 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:39.107169 25240 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.107533 22806 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:39.148085 25285 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.150842 25244 raft_consensus.cc:3055] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:39.151283 25244 raft_consensus.cc:740] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:39.109417 24873 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:39.109460 25276 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:39.151679 25041 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.040942 25196 raft_consensus.cc:515] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:39.152311 25276 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.108683 25250 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.053944 25266 consensus_queue.cc:1048] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.041s
I20251026 14:59:39.061600 25121 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:39.154167 25206 log.cc:927] Time spent T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 2.020s user 0.027s sys 0.021s
I20251026 14:59:39.154628 25196 leader_election.cc:290] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:39.154704 25276 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.014043 25259 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.155730 25121 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.111183 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:39.138944 22656 raft_consensus.cc:1275] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251026 14:59:39.099263 25253 consensus_queue.cc:237] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.141681 22622 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:39.142912 25283 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:39.144508 25254 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:39.093691 24843 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.145434 25269 raft_consensus.cc:2749] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:39.145368 25265 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.138741 25233 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:39.147058 25261 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:39.118371 25186 raft_consensus.cc:697] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 8 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:39.081079 25246 consensus_queue.cc:237] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.151643 22651 raft_consensus.cc:1275] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:39.157584 25255 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.158392 22657 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:39.159085 25193 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:39.160804 22802 raft_consensus.cc:2468] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
W20251026 14:59:39.133831 23369 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:39.198025 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:39.198894 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:39.198076 23404 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:39.162246 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.168799 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.201457 22728 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:39.201617 22732 raft_consensus.cc:2468] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:39.202450 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.203073 22728 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
W20251026 14:59:39.169553 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 46172us
W20251026 14:59:39.147360 25278 log.cc:927] Time spent T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20251026 14:59:39.171514 22622 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:39.092972 25281 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:39.172864 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:39.187888 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 46995us
I20251026 14:59:39.150408 25285 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.188548 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.140452 25243 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:39.191306 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.204037 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.206712 22622 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:39.170465 25195 raft_consensus.cc:697] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:39.213393 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.213716 22734 raft_consensus.cc:2393] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 10: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:39.291956 25290 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:39.298518 25290 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:39.299077 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.299707 22734 raft_consensus.cc:2468] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:39.300561 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.300688 25290 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:39.213912 25281 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:39.282066 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.301635 25281 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:39.302311 22732 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:39.174008 25252 raft_consensus.cc:2804] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:39.303755 22730 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
W20251026 14:59:39.283615 22771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 85158.1us
I20251026 14:59:39.318586 22657 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:39.319576 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.215121 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.320695 22733 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:39.311225 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.304550 22725 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:39.313848 22697 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:39.174310 25259 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.320263 22727 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:39.183485 25254 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.152074 25244 consensus_queue.cc:260] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:39.329560 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:39.242625 25245 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:39.242301 24843 raft_consensus.cc:2804] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:39.334718 24527 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.342945 25186 consensus_queue.cc:237] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.344398 25273 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.335182 22697 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:39.313581 25252 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:39.339715 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.353112 23404 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:39.342581 25283 raft_consensus.cc:2749] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:39.355355 24877 raft_consensus.cc:2749] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251026 14:59:39.355079 25195 consensus_queue.cc:237] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.355070 22654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1479) took 2139 ms. Trace:
I20251026 14:59:39.358441 22654 rpcz_store.cc:276] 1026 14:59:37.215840 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:37.215991 (+ 151us) service_pool.cc:225] Handling call
1026 14:59:37.216334 (+ 343us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:37.216688 (+ 354us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:37.216717 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:37.217180 (+ 463us) log.cc:844] Serialized 26 byte log entry
1026 14:59:37.286009 (+ 68829us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:37.286053 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:37.286126 (+ 73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:39.354051 (+2067925us) spinlock_profiling.cc:243] Waited 2.43 ms on lock 0x7b24003beb68. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8cdf065a4 00007fd8cdf046ab 00007fd8cdf93409 00007fd8cdf93095 00007fd8cdf925f0 00007fd8d0d26498 00007fd8d0d26bf8 00007fd8d0cfe438 00007fd8d0cf981a 00007fd8d0cf8c66
1026 14:59:39.354759 (+ 708us) raft_consensus.cc:1685] finished
1026 14:59:39.354782 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:39.355008 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1370,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":93,"spinlock_wait_cycles":5591424,"thread_start_us":68396,"threads_started":1,"wal-append.queue_time_us":210358}
I20251026 14:59:39.361090 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:39.361431 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:39.362358 22727 raft_consensus.cc:2410] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251026 14:59:39.362723 23404 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:39.346495 22725 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:39.364002 22728 raft_consensus.cc:2468] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:39.366302 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.367619 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:39.331161 25290 raft_consensus.cc:2749] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20251026 14:59:39.372403 22803 raft_consensus.cc:1275] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251026 14:59:39.373680 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:39.375216 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:39.376402 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:39.378167 22801 raft_consensus.cc:2368] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8.
I20251026 14:59:39.345590 24843 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:39.378734 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:39.379109 24843 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:39.379365 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:39.379519 25233 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:39.351023 25244 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:39.380438 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.381551 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.341257 25197 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:39.382246 22733 raft_consensus.cc:2468] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:39.321426 25243 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.379586 22807 raft_consensus.cc:2410] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251026 14:59:39.383553 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.378082 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:39.411021 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.418066 22729 raft_consensus.cc:2468] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:39.419051 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.350548 25252 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
W20251026 14:59:39.423267 22771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 89449.1us
I20251026 14:59:39.427735 22807 raft_consensus.cc:1275] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20251026 14:59:39.469084 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:39.471809 22801 pending_rounds.cc:85] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821: Aborting all ops after (but not including) 3
I20251026 14:59:39.472479 22801 pending_rounds.cc:107] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20251026 14:59:39.471778 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.472966 22801 raft_consensus.cc:2889] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:39.473987 22727 raft_consensus.cc:2393] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 8: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:39.477713 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:39.478582 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.478875 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:39.478915 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:39.480948 22651 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:39.481729 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:39.482049 22726 raft_consensus.cc:1240] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20251026 14:59:39.482818 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:39.484189 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:39.487001 22726 raft_consensus.cc:2393] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 5: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:39.490859 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:39.490929 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:39.490370 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:39.492709 22651 raft_consensus.cc:2468] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:39.434187 25215 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:39.498346 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.496906 22651 raft_consensus.cc:1275] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:39.497213 22657 raft_consensus.cc:1275] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20251026 14:59:39.498626 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 291992us
I20251026 14:59:39.345398 25245 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:39.502934 22622 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:39.504180 25245 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:39.504274 22726 raft_consensus.cc:1217] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[5.3-9.5] Dedup: 5.3->[7.4-9.5]
I20251026 14:59:39.502288 22651 raft_consensus.cc:1240] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:39.521733 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:39.522253 22725 raft_consensus.cc:1275] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:39.523293 24877 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:39.523748 25041 raft_consensus.cc:2804] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:39.524294 25041 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:39.524716 25045 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:39.525311 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.525997 22734 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 7.
I20251026 14:59:39.526101 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:39.526402 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.527076 22731 raft_consensus.cc:2468] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 3.
I20251026 14:59:39.527776 25171 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:39.528293 25171 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.521840 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:39.530550 22697 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:39.523840 24877 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.531586 22727 raft_consensus.cc:1240] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:39.532110 22726 raft_consensus.cc:1240] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 9. Current term is 10. Ops: [5.3-9.5]
I20251026 14:59:39.535012 22730 raft_consensus.cc:1240] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:39.428385 25197 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.538201 25255 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:39.525339 25045 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.539242 25197 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:39.539400 25253 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:39.539741 25259 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:39.540587 25045 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.540899 25259 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.541389 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.541769 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.542035 22725 raft_consensus.cc:2468] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 3.
I20251026 14:59:39.538446 25171 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.544361 25171 raft_consensus.cc:2749] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:39.544525 24877 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.542908 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.546401 22730 raft_consensus.cc:1240] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20251026 14:59:39.546877 22733 raft_consensus.cc:1275] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:39.541421 22731 raft_consensus.cc:1275] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251026 14:59:39.538954 25255 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.540364 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.548557 22732 raft_consensus.cc:2468] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
W20251026 14:59:39.544703 25231 consensus_meta.cc:287] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.526s user 0.004s sys 0.003s
I20251026 14:59:39.554358 22651 raft_consensus.cc:1275] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251026 14:59:39.549917 22727 raft_consensus.cc:1275] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:39.549079 22697 leader_election.cc:400] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8.
I20251026 14:59:39.538293 22729 raft_consensus.cc:1240] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:39.555575 22697 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:39.542899 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.556962 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:39.558048 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:39.558306 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.558473 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.558652 22652 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:39.560052 22808 raft_consensus.cc:2410] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20251026 14:59:39.524694 25041 raft_consensus.cc:3060] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:39.564090 22732 raft_consensus.cc:1240] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:39.565527 23369 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:39.566316 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:39.566799 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 63926.3us
W20251026 14:59:39.575917 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:39.576246 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.573067 22622 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:39.543038 25259 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.587782 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:39.588762 22652 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 7.
W20251026 14:59:39.562040 25152 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.668s user 0.012s sys 0.022s
I20251026 14:59:39.575728 25237 raft_consensus.cc:2804] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:39.548322 25215 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.539944 25253 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:39.616302 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 53215.4us
I20251026 14:59:39.574635 25287 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:39.617087 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 44113us
I20251026 14:59:39.612118 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.626978 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:39.593613 25152 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.732s user 0.012s sys 0.023s
I20251026 14:59:39.562477 25290 consensus_queue.cc:1048] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20251026 14:59:39.639317 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.570760 25291 raft_consensus.cc:2749] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251026 14:59:39.571238 24877 consensus_queue.cc:1048] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
I20251026 14:59:39.573416 25239 consensus_queue.cc:1048] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20251026 14:59:39.730970 22622 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
W20251026 14:59:39.739362 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 99966us
I20251026 14:59:39.740180 22733 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:39.740406 22624 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:39.676957 22659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1451) took 3542 ms (client timeout 2039 ms). Trace:
I20251026 14:59:39.729033 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:39.741110 22659 rpcz_store.cc:269] 1026 14:59:36.134972 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:36.166553 (+ 31581us) service_pool.cc:225] Handling call
1026 14:59:39.676763 (+3510210us) spinlock_profiling.cc:243] Waited 1.91 s on lock 0x7b540012f2d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:39.676883 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4394072320}
I20251026 14:59:39.696650 23404 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
W20251026 14:59:39.724766 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:39.643610 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:39.746752 22774 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:39.746922 22730 raft_consensus.cc:2468] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:39.750598 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.751403 25215 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:39.752164 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.753073 25266 consensus_queue.cc:1048] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 14:59:39.755177 25211 consensus_queue.cc:1048] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 14:59:39.753440 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:39.765376 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 199885us
I20251026 14:59:39.766359 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.755822 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:39.768020 23369 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:39.778772 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 47914.5us
I20251026 14:59:39.779834 22622 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:39.641644 25237 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:39.790484 22803 raft_consensus.cc:1240] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:39.727519 25259 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:39.795117 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.703967 25254 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.749506 25152 raft_consensus.cc:697] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:39.792683 25237 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:39.814482 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.817770 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.820008 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.755779 25245 raft_consensus.cc:2804] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:39.774675 25273 raft_consensus.cc:2804] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:39.829053 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.787300 25058 raft_consensus.cc:2749] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:39.830857 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.834772 22806 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:39.774107 25238 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:39.793910 25233 raft_consensus.cc:2804] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:39.840333 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.842926 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.843441 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.818140 25240 raft_consensus.cc:2749] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:39.653648 25287 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.846637 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.791687 24843 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.853417 22806 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
I20251026 14:59:39.856671 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.836369 25273 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:39.865024 22624 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:39.870257 22799 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:39.830636 25152 consensus_queue.cc:237] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.879765 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.883564 22624 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:39.871448 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.872303 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.886101 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.875456 23369 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:39.878839 22807 raft_consensus.cc:2410] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20251026 14:59:39.879763 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:39.879940 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:39.887135 22801 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:39.886466 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.882452 22799 raft_consensus.cc:2468] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:39.891153 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.892051 22800 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:39.892155 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.889467 22624 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:39.877090 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:39.889982 25246 raft_consensus.cc:2804] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:39.924825 25246 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:39.849676 25255 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.922318 25159 raft_consensus.cc:2804] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:39.925693 25159 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:39.926152 25159 raft_consensus.cc:3060] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:39.890902 25058 raft_consensus.cc:2804] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:39.928968 25287 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:39.929430 25058 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:39.929913 25058 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:39.923256 25265 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.925349 25246 raft_consensus.cc:3060] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:39.825139 25259 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:39.843923 25238 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:39.881336 25273 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:39.943445 22804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44594 (request call id 1337) took 1030 ms. Trace:
I20251026 14:59:39.945192 22805 raft_consensus.cc:1275] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20251026 14:59:39.965488 22805 pending_rounds.cc:85] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821: Aborting all ops after (but not including) 3
I20251026 14:59:39.965750 22805 pending_rounds.cc:107] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20251026 14:59:39.966084 22805 raft_consensus.cc:2889] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:39.941587 22806 raft_consensus.cc:1240] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:39.964763 22804 rpcz_store.cc:276] 1026 14:59:38.913386 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:38.914237 (+ 851us) service_pool.cc:225] Handling call
1026 14:59:39.943217 (+1028980us) spinlock_profiling.cc:243] Waited 96.5 ms on lock 0x7b540020b7d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:39.943366 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":221995776}
I20251026 14:59:39.954190 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:39.973536 22801 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
I20251026 14:59:39.938917 25236 raft_consensus.cc:2804] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:39.940145 25243 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.985460 22800 raft_consensus.cc:1275] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:39.935410 22624 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:40.014040 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:40.014225 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 78907us
I20251026 14:59:40.015254 22624 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:40.017038 22624 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:40.023974 22731 raft_consensus.cc:3055] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:40.024556 22731 raft_consensus.cc:740] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:40.024672 22624 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:40.029176 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 249281us
I20251026 14:59:40.029331 22729 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:39.966509 25265 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:40.025578 22731 consensus_queue.cc:260] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.031605 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:40.031324 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:40.050526 22731 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
W20251026 14:59:39.987886 25252 consensus_meta.cc:287] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.517s user 0.012s sys 0.000s
I20251026 14:59:40.053131 22622 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:40.030721 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:40.043824 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:39.988379 25236 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:40.059440 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:40.029026 25239 raft_consensus.cc:2804] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:40.050707 25243 raft_consensus.cc:2804] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:40.072710 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:40.031494 25276 raft_consensus.cc:2804] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:40.065470 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:40.050446 25124 raft_consensus.cc:2804] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:39.995353 25237 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.068186 25239 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:40.092222 25239 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:40.092351 25237 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:39.989567 25253 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:40.084877 25124 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:40.055330 25265 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.087350 25315 raft_consensus.cc:2804] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:40.089133 25259 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:40.090523 22655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1498) took 1191 ms. Trace:
I20251026 14:59:40.095988 22659 raft_consensus.cc:2468] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:40.096560 22731 raft_consensus.cc:1275] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20251026 14:59:40.118721 22655 rpcz_store.cc:276] 1026 14:59:38.898880 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:39.013743 (+114863us) service_pool.cc:225] Handling call
1026 14:59:40.090249 (+1076506us) spinlock_profiling.cc:243] Waited 2.37 ms on lock 0x7b5400157f54. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:40.090414 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5446016}
I20251026 14:59:40.119207 22731 pending_rounds.cc:85] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e: Aborting all ops after (but not including) 3
I20251026 14:59:40.119721 22731 pending_rounds.cc:107] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20251026 14:59:40.120005 22731 raft_consensus.cc:2889] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:40.099985 22652 raft_consensus.cc:1240] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:40.069442 25236 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:40.093063 25276 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:40.072949 25243 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:40.124958 25281 consensus_meta.cc:287] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.823s user 0.002s sys 0.011s
I20251026 14:59:40.125911 22656 raft_consensus.cc:1275] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20251026 14:59:40.057880 25252 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:40.102500 25230 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s user 0.013s sys 0.049s
I20251026 14:59:40.120258 25315 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:40.126222 24843 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:40.150630 25243 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
W20251026 14:59:40.205981 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 107140us
I20251026 14:59:40.226692 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:40.239084 22622 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:40.250162 22622 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:40.257668 25259 raft_consensus.cc:2804] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:40.266117 22622 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:40.158254 25230 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s user 0.016s sys 0.052s
I20251026 14:59:40.151027 25124 raft_consensus.cc:3060] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:40.323550 22697 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:40.330488 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 64502.8us
I20251026 14:59:40.159386 25315 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:40.332664 22622 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:40.333568 22731 raft_consensus.cc:1275] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251026 14:59:40.206282 25276 raft_consensus.cc:3060] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
W20251026 14:59:40.350373 22697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:40.349927 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:40.357646 25309 log.cc:927] Time spent T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.167s user 0.001s sys 0.000s
I20251026 14:59:40.258131 25259 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:40.258716 25249 raft_consensus.cc:2749] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:40.155968 25281 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.331375 25238 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:40.424655 22731 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:40.347378 25237 raft_consensus.cc:2804] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20251026 14:59:40.444445 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:40.364058 25199 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:40.445741 25199 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.448454 25199 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:40.458016 25249 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:40.458819 25265 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:40.426069 25259 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:40.361541 25192 raft_consensus.cc:2804] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:40.462942 25250 raft_consensus.cc:2749] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:40.440356 25237 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:40.493083 25237 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
W20251026 14:59:40.495868 24849 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.010s sys 0.016s
W20251026 14:59:40.496274 24849 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.557s user 0.011s sys 0.016s
I20251026 14:59:40.526975 22731 raft_consensus.cc:1275] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:40.552124 22700 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:40.556773 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 224162us
I20251026 14:59:40.560209 22622 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:40.566708 22700 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:40.577838 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1483) took 1043 ms. Trace:
I20251026 14:59:40.578302 22734 rpcz_store.cc:276] 1026 14:59:39.533974 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:39.534239 (+ 265us) service_pool.cc:225] Handling call
1026 14:59:39.877579 (+343340us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:39.877883 (+ 304us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:39.877911 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:39.877952 (+ 41us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:39.877982 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:39.878049 (+ 67us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:40.577649 (+699600us) spinlock_profiling.cc:243] Waited 337 ms on lock 0x7b54000b8bd0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:40.577771 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8286,"raft.queue_time_us":2684,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":107,"spinlock_wait_cycles":774487936}
I20251026 14:59:40.579929 22802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49652 (request call id 302) took 1217 ms. Trace:
I20251026 14:59:40.580506 22802 rpcz_store.cc:276] 1026 14:59:39.362312 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:39.363987 (+ 1675us) service_pool.cc:225] Handling call
1026 14:59:40.578831 (+1214844us) spinlock_profiling.cc:243] Waited 2.02 ms on lock 0x7b5800002a98. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8ce074451 00007fd8ce074302 00007fd8d6f0dad9 00007fd8d6090d30 00007fd8d615984b 00007fd8d61598d1 00007fd8d611b580 00007fd8d6105ee9 00007fd8d6105992 00007fd8cf69fd34
1026 14:59:40.579128 (+ 297us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:40.579419 (+ 291us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:40.579462 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:40.579525 (+ 63us) raft_consensus.cc:1645] Marking committed up to 6
1026 14:59:40.579586 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:40.579669 (+ 83us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:40.579847 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4655872}
I20251026 14:59:40.581655 22658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1490) took 1713 ms. Trace:
I20251026 14:59:40.582047 22658 rpcz_store.cc:276] 1026 14:59:38.868245 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:38.870715 (+ 2470us) service_pool.cc:225] Handling call
1026 14:59:38.871036 (+ 321us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:39.110869 (+239833us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:40.581417 (+1470548us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b5400265554. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:40.581594 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":549568896}
W20251026 14:59:40.487443 25273 consensus_meta.cc:287] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.511s user 0.009s sys 0.001s
I20251026 14:59:40.586526 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:40.587235 22725 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 9.
I20251026 14:59:40.589743 22700 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:40.605899 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 45845.7us
I20251026 14:59:40.473912 25252 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:40.476714 25199 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:40.611897 25199 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:40.613067 25321 log.cc:927] Time spent T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20251026 14:59:40.458833 25249 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.614800 25267 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:40.615396 25267 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.616225 25249 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:40.616838 25250 raft_consensus.cc:2749] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:40.617411 25267 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:40.620361 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:40.482369 25236 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.625625 23370 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:40.626477 22651 raft_consensus.cc:2368] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7.
I20251026 14:59:40.626932 25095 raft_consensus.cc:2749] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:40.627310 25236 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:40.604346 25241 raft_consensus.cc:2804] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:40.634943 25241 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:40.635442 25241 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:40.563673 25192 raft_consensus.cc:697] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:40.653607 25192 consensus_queue.cc:237] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:40.662729 25281 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:40.668599 25159 consensus_meta.cc:287] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1: Time spent flushing consensus metadata: real 0.742s user 0.011s sys 0.000s
I20251026 14:59:40.669245 25159 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:40.587235 25212 raft_consensus.cc:2749] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:40.678570 22653 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff"
dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:46598
I20251026 14:59:40.679266 25233 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:40.680028 25233 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:40.682473 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:40.679438 22653 raft_consensus.cc:606] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 LEADER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:40.712519 22807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1234) took 1209 ms. Trace:
I20251026 14:59:40.685959 25245 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:40.698199 25246 consensus_meta.cc:287] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.761s user 0.010s sys 0.001s
W20251026 14:59:40.703338 25041 consensus_meta.cc:287] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 1.125s user 0.007s sys 0.006s
W20251026 14:59:40.704752 25058 consensus_meta.cc:287] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.774s user 0.008s sys 0.004s
I20251026 14:59:40.683352 25281 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:40.698630 25159 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:40.713081 22807 rpcz_store.cc:276] 1026 14:59:39.494610 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:39.566245 (+ 71635us) service_pool.cc:225] Handling call
1026 14:59:40.703615 (+1137370us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b54000c6554. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:40.703829 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":692723072}
I20251026 14:59:40.606343 25273 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.685215 25212 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:40.706921 25239 consensus_meta.cc:287] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.614s user 0.007s sys 0.004s
I20251026 14:59:40.698478 25252 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:40.683068 25237 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:40.724671 25245 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:40.759987 22653 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "143c213a935041868ad3b0d9584ad863"
dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:46598
I20251026 14:59:40.760927 22653 raft_consensus.cc:606] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:40.761382 22653 raft_consensus.cc:614] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251026 14:59:40.707476 25124 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.708920 25276 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.709693 25259 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.743191 25058 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.742726 25041 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.743077 25246 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.746697 25281 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.757131 25239 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.759006 25252 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.752915 25212 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.808359 22653 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "32b237245fe346a689e59ddc315b4577"
dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:46598
I20251026 14:59:40.835533 22653 raft_consensus.cc:606] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:40.835932 22653 raft_consensus.cc:614] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251026 14:59:40.811038 25239 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:40.834849 22650 raft_consensus.cc:2368] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7.
I20251026 14:59:40.842394 22805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1230) took 1444 ms. Trace:
I20251026 14:59:40.842885 22805 rpcz_store.cc:276] 1026 14:59:39.397637 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:39.422435 (+ 24798us) service_pool.cc:225] Handling call
1026 14:59:39.944718 (+522283us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:39.966573 (+ 21855us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:40.842193 (+875620us) spinlock_profiling.cc:243] Waited 522 ms on lock 0x7b540004bad0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:40.842332 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1200240512}
I20251026 14:59:40.854897 25058 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:40.861305 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 271628us
I20251026 14:59:40.862107 22700 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:40.866989 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:40.869807 22651 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570"
dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:46598
I20251026 14:59:40.870465 22651 raft_consensus.cc:606] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:40.870846 22651 raft_consensus.cc:614] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251026 14:59:40.871644 22700 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:40.874572 22700 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:40.876962 25243 consensus_meta.cc:287] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.557s user 0.009s sys 0.004s
I20251026 14:59:40.871552 25158 raft_consensus.cc:2749] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:40.875931 24849 raft_consensus.cc:2749] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251026 14:59:40.903630 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 150768us
I20251026 14:59:40.882823 25253 raft_consensus.cc:2749] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:40.910876 22651 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663"
dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:46598
I20251026 14:59:40.911571 22651 raft_consensus.cc:606] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:40.911926 22651 raft_consensus.cc:614] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251026 14:59:40.902521 25336 raft_consensus.cc:2804] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:40.907217 25237 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Requested vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:40.917495 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20251026 14:59:40.932600 22654 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 1263) took 1535 ms (client timeout 1931 ms). Trace:
W20251026 14:59:40.933002 22654 rpcz_store.cc:269] 1026 14:59:39.396932 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:39.468662 (+ 71730us) service_pool.cc:225] Handling call
1026 14:59:40.932392 (+1463730us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54001fc554. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:40.932536 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2915967616}
W20251026 14:59:40.938488 25333 log.cc:927] Time spent T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
W20251026 14:59:40.926726 25315 consensus_meta.cc:287] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.564s user 0.013s sys 0.000s
I20251026 14:59:40.939121 25315 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.940402 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:59:40.941395 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:59:40.942688 25276 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:40.914564 25273 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:40.945444 25281 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:40.945878 25212 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:40.946731 25315 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:40.947741 25241 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.918622 25336 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:40.912927 25332 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:59:40.958803 22604 auto_rebalancer.cc:254] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
I20251026 14:59:40.901886 25243 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.911675 24849 consensus_queue.cc:1048] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20251026 14:59:40.934218 25259 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:40.962926 25341 log.cc:927] Time spent T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.059s user 0.008s sys 0.000s
I20251026 14:59:40.954617 25246 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:40.967926 25327 log.cc:927] Time spent T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.121s user 0.001s sys 0.000s
I20251026 14:59:40.955117 25041 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:40.974576 25243 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:40.966857 25336 raft_consensus.cc:3060] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:40.988102 25241 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:40.989729 22604 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 129989aafdc84d6b8e56fe97c7182fff, TS 0d684445287b450a8f738b6ad4b59821 -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:41.031921 22604 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 129989aafdc84d6b8e56fe97c7182fff, TS 0d684445287b450a8f738b6ad4b59821 -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.969641 25252 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:41.034852 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 121599us
W20251026 14:59:41.035396 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 85017.1us
W20251026 14:59:41.036222 22697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:41.037174 22700 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:40.998160 25240 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:40.969204 25332 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:41.040285 22604 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 1badbeef1a1c4427bbfa1b762c7d0739, TS 562bb1e1c8e445b8baecd29c3669658e -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:41.042759 22604 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 1badbeef1a1c4427bbfa1b762c7d0739, TS 562bb1e1c8e445b8baecd29c3669658e -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:41.038947 25241 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:41.045373 25199 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:41.042174 25240 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:41.046916 25252 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:41.047508 25252 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:41.044600 25241 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:40.979643 25245 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:41.059219 25344 consensus_queue.cc:1048] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20251026 14:59:41.059993 25344 consensus_queue.cc:1243] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer df3a0d1619ee48e8ab68ee0543a72a29 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20251026 14:59:41.060448 25315 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:41.061193 25315 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:41.063319 25315 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:41.065794 25272 raft_consensus.cc:2804] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:41.066751 25272 raft_consensus.cc:697] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 8 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
W20251026 14:59:41.067149 25326 log.cc:927] Time spent T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.569s user 0.004s sys 0.000s
I20251026 14:59:41.067836 25272 consensus_queue.cc:237] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:41.065932 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 1.29894e+06us
I20251026 14:59:41.040689 25192 raft_consensus.cc:2804] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:41.073899 25192 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:41.075407 25192 raft_consensus.cc:3060] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:41.077953 22658 raft_consensus.cc:1240] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:41.078568 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:41.079731 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:41.080021 22650 raft_consensus.cc:2368] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 8. Current term is 9.
I20251026 14:59:41.080436 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:41.081508 22651 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
W20251026 14:59:41.082809 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:41.080518 22654 raft_consensus.cc:2410] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20251026 14:59:41.086651 23404 raft_consensus.cc:1240] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 8 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:41.086683 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:41.088969 22651 raft_consensus.cc:1240] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:41.091866 22651 raft_consensus.cc:1275] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:41.088738 25233 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:41.096264 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:41.098115 22651 raft_consensus.cc:2410] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20251026 14:59:41.100912 22654 raft_consensus.cc:1240] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:41.103040 22650 raft_consensus.cc:1275] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251026 14:59:41.107605 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:41.111061 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:41.115435 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:41.116856 22654 raft_consensus.cc:2410] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20251026 14:59:41.094111 22624 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:41.121455 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:41.122545 22657 raft_consensus.cc:2393] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 7: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:41.128793 22650 pending_rounds.cc:85] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29: Aborting all ops after (but not including) 1
I20251026 14:59:41.129398 22650 pending_rounds.cc:107] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20251026 14:59:41.129885 22650 raft_consensus.cc:2889] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:41.139559 22653 raft_consensus.cc:1240] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:41.141263 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:41.088804 25186 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:41.145787 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:41.146296 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:41.149082 22652 raft_consensus.cc:2393] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 7: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
W20251026 14:59:41.149255 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 55287.7us
I20251026 14:59:41.089934 25285 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:41.137593 24849 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:41.204463 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:41.204983 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:41.207726 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:41.209851 22802 raft_consensus.cc:1275] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20251026 14:59:41.137740 25253 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:41.159879 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:41.159170 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:41.157838 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:41.228673 22733 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:41.232437 22727 raft_consensus.cc:2410] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20251026 14:59:41.160580 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 91064.8us
I20251026 14:59:41.160274 22731 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:41.163674 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:41.235141 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:41.235740 22730 raft_consensus.cc:2468] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:41.236559 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:41.237039 22730 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 9
I20251026 14:59:41.159742 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:41.196226 25345 log.cc:927] Time spent T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.111s user 0.000s sys 0.000s
I20251026 14:59:41.196983 25041 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:41.197433 25186 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:41.199333 25336 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:41.202669 25281 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:41.159359 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:41.215723 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:41.216965 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:41.221110 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:41.225543 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:41.255455 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:41.256341 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:41.270273 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:41.259169 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:41.260344 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:41.203363 25236 raft_consensus.cc:2804] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:41.276168 22799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1239) took 1744 ms. Trace:
I20251026 14:59:41.285205 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:41.278415 22808 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:41.286710 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:41.276309 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:41.279680 22727 raft_consensus.cc:2393] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 7: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:41.287431 22734 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:41.277565 22803 raft_consensus.cc:2468] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:41.147053 25252 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:41.276809 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:41.290640 22800 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:41.291729 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:41.292297 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:41.292837 22805 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:41.299904 22802 raft_consensus.cc:1275] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20251026 14:59:41.296819 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:41.303772 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:41.303859 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:41.304280 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:41.305258 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:41.264653 24740 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:41.260959 25343 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:41.307693 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:41.307917 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:41.310379 22652 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:41.310536 22732 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 8
I20251026 14:59:41.315979 22731 raft_consensus.cc:1275] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20251026 14:59:41.281055 25199 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:41.321249 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:41.328758 22806 raft_consensus.cc:2468] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:41.230402 25332 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:41.322533 22800 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:41.281033 25264 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:41.275403 25288 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:41.308566 25344 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:41.326436 22771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 71313.1us
I20251026 14:59:41.328075 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:41.342085 22801 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:41.265326 25231 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:41.276885 25124 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:41.285339 25236 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:41.227453 25285 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:41.330098 22734 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:41.321844 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:41.233402 24849 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:41.335256 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:41.364704 22729 raft_consensus.cc:2393] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 7: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:41.365762 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:41.255402 25253 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:41.372236 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:41.354496 25236 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
W20251026 14:59:41.325872 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 92110.5us
I20251026 14:59:41.332791 25199 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:41.384683 22554 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50170 (request call id 61) took 12793 ms (client timeout 15000 ms). Trace:
I20251026 14:59:41.283324 25041 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:41.384965 22728 raft_consensus.cc:3060] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:41.284906 25281 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:41.355099 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:41.390161 22734 raft_consensus.cc:2393] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 6: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:41.355381 22725 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:41.358903 22652 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 8.
I20251026 14:59:41.341077 22730 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
I20251026 14:59:41.366570 22806 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 7
I20251026 14:59:41.393577 22733 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:41.393443 23370 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:41.336723 25245 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:41.327772 24740 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:41.360936 22732 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:41.329655 25343 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:41.338567 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:41.405057 22731 raft_consensus.cc:3060] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:41.405790 22728 raft_consensus.cc:2468] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 5.
I20251026 14:59:41.350436 25215 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:41.412052 25199 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:41.339254 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:41.413714 22729 raft_consensus.cc:2393] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 8: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:41.340313 25264 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:41.413015 22800 raft_consensus.cc:2468] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:41.401376 25307 raft_consensus.cc:2804] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:41.403512 22774 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:41.434453 22774 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:41.353034 25344 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:41.396267 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:41.345397 25288 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:41.354725 25231 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:41.418941 22806 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:41.437294 22734 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:41.387202 25240 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:41.404074 25197 consensus_queue.cc:1048] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20251026 14:59:41.385176 22554 rpcz_store.cc:269] 1026 14:59:28.596480 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:28.601749 (+ 5269us) service_pool.cc:225] Handling call
1026 14:59:41.384586 (+12782837us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 14:59:36.883755 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 14:59:36.898201 (+ 14446us) write_op.cc:432] Acquiring schema lock in shared mode
1026 14:59:36.898261 (+ 60us) write_op.cc:435] Acquired schema lock
1026 14:59:36.898292 (+ 31us) tablet.cc:660] Decoding operations
1026 14:59:36.899054 (+ 762us) write_op.cc:620] Acquiring the partition lock for write op
1026 14:59:36.899168 (+ 114us) write_op.cc:641] Partition lock acquired for write op
1026 14:59:36.899212 (+ 44us) tablet.cc:683] Acquiring locks for 3 operations
1026 14:59:36.899653 (+ 441us) tablet.cc:699] Row locks acquired
1026 14:59:36.899678 (+ 25us) write_op.cc:260] PREPARE: finished
1026 14:59:36.899884 (+ 206us) write_op.cc:270] Start()
1026 14:59:36.900036 (+ 152us) write_op.cc:276] Timestamp: P: 1761490776899810 usec, L: 0
1026 14:59:36.900067 (+ 31us) op_driver.cc:348] REPLICATION: starting
1026 14:59:36.933153 (+ 33086us) log.cc:844] Serialized 1334 byte log entry
1026 14:59:37.149946 (+216793us) op_driver.cc:464] REPLICATION: finished
1026 14:59:37.353567 (+203621us) spinlock_profiling.cc:243] Waited 21.7 ms on lock 0x7b440002304c. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8cdfc0905 00007fd8cdfc1322 00007fd8d0d0a231 00007fd8d0d05b5a 00007fd8ce09459c 00007fd8d6e6d219 00007fd8ce0b0c4b 00007fd8ce0aeeef 00007fd8ce0b11b2 00007fd8d4b8c650
1026 14:59:39.336195 (+1982628us) spinlock_profiling.cc:243] Waited 3.75 ms on lock 0x7b1800267d00. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce0b1992 00007fd8ce0b3d61 00007fd8ce0b3d19 00007fd8ce0b3ca9 00007fd8ce0b3c71 00007fd8ce0b2f6d 0000000000379254 0000000000379099 00007fd8ce094d1a
1026 14:59:39.336411 (+ 216us) write_op.cc:301] APPLY: starting
1026 14:59:39.336495 (+ 84us) tablet.cc:1365] starting BulkCheckPresence
1026 14:59:39.336645 (+ 150us) tablet.cc:1368] finished BulkCheckPresence
1026 14:59:39.336677 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 14:59:39.337205 (+ 528us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 14:59:39.374155 (+ 36950us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
1026 14:59:39.398815 (+ 24660us) write_op.cc:312] APPLY: finished
1026 14:59:39.480019 (+ 81204us) log.cc:844] Serialized 34 byte log entry
1026 14:59:40.352586 (+872567us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:40.369550 (+ 16964us) write_op.cc:454] Released schema lock
1026 14:59:40.415957 (+ 46407us) write_op.cc:341] FINISH: Updating metrics
1026 14:59:40.492110 (+ 76153us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:40.492162 (+ 52us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":606138,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":1069948,"apply.run_cpu_time_us":82206,"apply.run_wall_time_us":1104453,"num_ops":3,"prepare.queue_time_us":140502,"prepare.run_cpu_time_us":19622,"prepare.run_wall_time_us":152922,"raft.queue_time_us":65472,"raft.run_cpu_time_us":42268,"raft.run_wall_time_us":1125406,"replication_time_us":249718,"spinlock_wait_cycles":58525440,"thread_start_us":2215816,"threads_started":5,"wal-append.queue_time_us":874421,"wal-append.run_cpu_time_us":20722,"wal-append.run_wall_time_us":2211708}]]}
I20251026 14:59:41.406031 25233 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:41.412515 25215 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:41.435302 22731 raft_consensus.cc:2468] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:41.543274 22801 raft_consensus.cc:2468] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:41.419180 24873 consensus_queue.cc:1048] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.247s
I20251026 14:59:41.424837 22725 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
I20251026 14:59:41.665627 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:59:41.522598 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 72759.5us
I20251026 14:59:41.441043 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:41.666566 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:41.668166 22774 leader_election.cc:400] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7.
I20251026 14:59:41.668274 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:41.668485 22774 leader_election.cc:403] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20251026 14:59:41.519454 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:41.670444 22774 leader_election.cc:400] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7.
I20251026 14:59:41.670830 22774 leader_election.cc:403] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20251026 14:59:41.669538 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:41.671623 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:41.672326 22731 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:41.673147 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:41.674445 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:41.675354 22731 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:41.469169 22730 raft_consensus.cc:1275] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20251026 14:59:41.699193 22806 raft_consensus.cc:3060] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:41.502754 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:41.679365 22733 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
W20251026 14:59:41.699909 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 354227us
I20251026 14:59:41.541781 22734 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:41.285575 22799 rpcz_store.cc:276] 1026 14:59:39.531983 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:39.901356 (+369373us) service_pool.cc:225] Handling call
1026 14:59:39.901691 (+ 335us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:39.901941 (+ 250us) raft_consensus.cc:1534] Early marking committed up to index 5
1026 14:59:40.319807 (+417866us) log.cc:844] Serialized 14 byte log entry
1026 14:59:41.135639 (+815832us) log.cc:844] Serialized 14 byte log entry
1026 14:59:41.275670 (+140031us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:41.275748 (+ 78us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:41.275822 (+ 74us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:41.275921 (+ 99us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:41.276092 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":993112,"prepare.queue_time_us":231733,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":76,"raft.queue_time_us":707698,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":6296,"spinlock_wait_cycles":33120128,"thread_start_us":472701,"threads_started":2,"wal-append.queue_time_us":129517}
I20251026 14:59:41.669524 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:41.632910 22697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:41.687659 22805 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:41.412111 25343 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:41.676975 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:41.701387 22800 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:41.669785 25186 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:41.704404 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:41.718849 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:41.718695 22805 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 10.
I20251026 14:59:41.712481 22725 raft_consensus.cc:2393] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 8: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
W20251026 14:59:41.497795 25335 log.cc:927] Time spent T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.632s user 0.001s sys 0.011s
I20251026 14:59:41.720518 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:41.703147 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:41.717023 22658 raft_consensus.cc:1240] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20251026 14:59:41.710563 22650 raft_consensus.cc:1240] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:41.721096 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:41.677819 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:41.703531 22729 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:41.732385 22800 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:41.724351 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:41.724794 22657 raft_consensus.cc:1240] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 8. Current term is 9. Ops: []
I20251026 14:59:41.500656 25241 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:41.439545 25348 raft_consensus.cc:2804] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:41.491771 25231 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:41.723397 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:41.736711 22799 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:41.738623 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:41.667512 24849 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:41.520639 25336 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:41.497500 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:41.439319 25196 raft_consensus.cc:2804] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:41.495569 25344 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:41.751133 22655 raft_consensus.cc:1240] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20251026 14:59:41.541096 25194 consensus_queue.cc:1048] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.210s
I20251026 14:59:41.752377 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:41.754243 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:41.754490 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:41.754312 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:41.807446 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:41.492940 25288 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:41.755136 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:41.816531 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:41.775743 22800 raft_consensus.cc:2468] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:41.762066 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:41.818372 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:41.819455 22725 raft_consensus.cc:2410] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251026 14:59:41.766551 22658 raft_consensus.cc:1275] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:41.784793 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:41.455847 25263 consensus_queue.cc:1059] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.284s
I20251026 14:59:41.681699 25245 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:41.756299 25348 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:41.789736 22734 raft_consensus.cc:1275] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251026 14:59:41.790917 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:41.704319 25214 raft_consensus.cc:2749] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7.
I20251026 14:59:41.648186 25226 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:41.798187 22651 raft_consensus.cc:1275] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:41.706002 25285 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:41.796967 22806 raft_consensus.cc:2468] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:41.726205 25343 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:41.861174 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:41.710073 24740 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:41.712491 25307 raft_consensus.cc:697] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 8 LEADER]: Becoming Leader. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:41.808643 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:41.456471 25185 consensus_queue.cc:1048] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.330s
I20251026 14:59:41.746443 25193 raft_consensus.cc:2749] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7.
I20251026 14:59:41.821592 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:41.875984 22803 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:41.876417 25307 consensus_queue.cc:237] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:41.879177 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:41.885933 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:41.886972 22804 raft_consensus.cc:2368] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 9. Current term is 10.
I20251026 14:59:41.887924 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:41.826396 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:41.786892 22657 raft_consensus.cc:1275] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251026 14:59:41.833508 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:41.889997 22725 raft_consensus.cc:1275] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:41.889052 22803 raft_consensus.cc:2468] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:41.837615 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:41.855020 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:41.796445 25281 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:41.859202 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:41.891753 22803 raft_consensus.cc:1275] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20251026 14:59:41.892113 22803 pending_rounds.cc:85] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821: Aborting all ops after (but not including) 3
I20251026 14:59:41.798429 25196 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:41.862551 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:41.892333 22803 pending_rounds.cc:107] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20251026 14:59:41.892467 22806 raft_consensus.cc:1240] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:41.892650 22803 raft_consensus.cc:2889] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:41.867193 25226 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:41.871250 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:41.861330 25257 consensus_queue.cc:1059] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20251026 14:59:41.871811 25343 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:41.847553 25352 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:41.867471 25272 consensus_queue.cc:1059] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20251026 14:59:41.873766 25185 raft_consensus.cc:3055] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:41.893615 22806 raft_consensus.cc:1240] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 8 LEADER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:41.872956 25263 consensus_queue.cc:1048] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20251026 14:59:41.907042 22734 raft_consensus.cc:1275] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251026 14:59:41.908388 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:41.914606 22805 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:41.845688 25348 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:41.858268 25041 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:41.925185 25158 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:41.856729 25253 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:41.829546 25186 consensus_queue.cc:1059] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20251026 14:59:41.925773 22697 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:41.836512 25144 consensus_queue.cc:1048] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.060s
I20251026 14:59:41.928984 22806 raft_consensus.cc:1240] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20251026 14:59:41.904012 22801 raft_consensus.cc:2393] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 7: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:41.845453 25288 consensus_queue.cc:1059] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20251026 14:59:41.935112 22725 raft_consensus.cc:3055] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:41.979969 22725 raft_consensus.cc:740] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:41.980516 22622 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:41.939188 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:41.935629 22624 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:41.981621 22658 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:41.981969 22700 leader_election.cc:400] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 8. Current term is 9.
I20251026 14:59:41.948724 22732 raft_consensus.cc:2468] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 4.
I20251026 14:59:41.954624 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:41.864357 25264 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:41.918967 25257 raft_consensus.cc:3055] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 8 LEADER]: Stepping down as leader of term 8
I20251026 14:59:42.001355 25257 raft_consensus.cc:740] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:42.002527 25257 consensus_queue.cc:260] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:42.003734 25343 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:42.004050 25257 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:41.905169 25196 raft_consensus.cc:3060] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
W20251026 14:59:42.007303 22604 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 7de11d91eebd4cd0b3183e4c044d4004, TS 0d684445287b450a8f738b6ad4b59821 -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:42.007786 22604 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 7de11d91eebd4cd0b3183e4c044d4004, TS 0d684445287b450a8f738b6ad4b59821 -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:41.978626 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 52932.5us
I20251026 14:59:41.855453 25245 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:41.927618 25185 raft_consensus.cc:740] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:41.946147 22651 raft_consensus.cc:1240] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:42.018240 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:42.019383 22651 raft_consensus.cc:2393] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 7: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:41.947496 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:41.983721 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:41.985008 22700 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:41.972929 25354 consensus_queue.cc:1059] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20251026 14:59:41.990530 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:41.980788 22725 consensus_queue.cc:260] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:42.023555 22725 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:42.023972 22697 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:42.023555 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:42.025101 22801 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:42.025223 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:41.986784 25351 log.cc:927] Time spent T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.446s user 0.001s sys 0.000s
I20251026 14:59:41.978580 25255 consensus_queue.cc:1048] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20251026 14:59:41.982842 25288 raft_consensus.cc:3055] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:41.972976 25195 raft_consensus.cc:3055] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:42.028024 22651 raft_consensus.cc:2410] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 11 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20251026 14:59:41.980558 25253 raft_consensus.cc:2764] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
W20251026 14:59:42.058873 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 240650us
I20251026 14:59:42.033721 25353 raft_consensus.cc:2804] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:41.968505 25352 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:42.035203 25226 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:42.016878 25363 log.cc:927] Time spent T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.137s user 0.001s sys 0.000s
I20251026 14:59:41.932456 25281 consensus_queue.cc:1059] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.057s
I20251026 14:59:42.022998 25273 raft_consensus.cc:2804] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:42.048074 25195 raft_consensus.cc:740] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
W20251026 14:59:42.053169 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 212539us
I20251026 14:59:42.055088 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:42.048723 25288 raft_consensus.cc:740] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:42.023214 25041 raft_consensus.cc:2749] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:42.023214 22658 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
I20251026 14:59:42.034078 25343 raft_consensus.cc:3055] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:41.977939 25158 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:42.064340 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:42.068979 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:42.065855 23367 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:42.067708 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:42.071651 22653 raft_consensus.cc:2393] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 8: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:42.060289 25353 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:42.074342 22658 raft_consensus.cc:2393] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 8: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:42.074607 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:42.075382 22651 raft_consensus.cc:2393] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:42.059671 25241 raft_consensus.cc:2804] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:42.078128 22725 raft_consensus.cc:1275] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20251026 14:59:42.090987 22725 pending_rounds.cc:85] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e: Aborting all ops after (but not including) 4
W20251026 14:59:42.080340 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 250726us
I20251026 14:59:42.061932 23403 raft_consensus.cc:1240] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:42.091353 22725 pending_rounds.cc:107] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20251026 14:59:42.096843 22725 raft_consensus.cc:2889] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:42.048453 24849 consensus_queue.cc:1059] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20251026 14:59:42.051815 25354 consensus_queue.cc:1048] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20251026 14:59:42.102034 22653 raft_consensus.cc:1240] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:42.099823 22774 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:42.067688 25215 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:42.134974 22773 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:42.073925 25273 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:42.075207 25343 raft_consensus.cc:740] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:42.062170 25226 consensus_queue.cc:1048] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20251026 14:59:42.090823 25241 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:42.079598 25353 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:42.108923 25281 raft_consensus.cc:3055] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:42.066850 25253 consensus_queue.cc:1059] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.066s
I20251026 14:59:42.130777 25214 raft_consensus.cc:2749] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:42.097028 25216 consensus_queue.cc:1059] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20251026 14:59:42.077617 25264 consensus_queue.cc:1059] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.075s
I20251026 14:59:42.119836 25307 raft_consensus.cc:2749] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251026 14:59:42.152979 22773 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:42.151747 25273 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:42.109443 24740 consensus_queue.cc:1048] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20251026 14:59:42.077693 25185 consensus_queue.cc:260] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:42.157667 25285 consensus_queue.cc:1048] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20251026 14:59:42.131165 25336 consensus_queue.cc:1059] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20251026 14:59:42.158855 25215 raft_consensus.cc:2749] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:42.195170 22799 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
I20251026 14:59:42.109069 25186 consensus_queue.cc:1048] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.068s
W20251026 14:59:42.204852 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 194647us
I20251026 14:59:42.141367 25193 consensus_queue.cc:1048] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20251026 14:59:42.108986 25231 consensus_queue.cc:1048] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.067s
I20251026 14:59:42.171708 25281 raft_consensus.cc:740] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:42.162349 25241 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
W20251026 14:59:42.204614 25192 consensus_meta.cc:287] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 1.124s user 0.012s sys 0.002s
I20251026 14:59:42.251219 25192 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:42.202162 25255 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:42.224294 22622 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:42.253576 25255 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:42.253993 22622 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:42.161590 25195 consensus_queue.cc:260] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:42.255424 25195 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:42.255896 25255 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:42.256495 25255 raft_consensus.cc:2804] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:42.256834 25255 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
W20251026 14:59:42.232260 25359 log.cc:927] Time spent T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.176s user 0.000s sys 0.001s
I20251026 14:59:42.257584 22727 raft_consensus.cc:2393] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 6: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:42.232108 25281 consensus_queue.cc:260] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:42.258888 25281 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:42.237085 25196 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:42.262094 22805 raft_consensus.cc:2393] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 6: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:42.262923 25196 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:42.161978 25288 consensus_queue.cc:260] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:42.264556 25288 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:42.245972 25369 raft_consensus.cc:3055] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:42.265470 25369 raft_consensus.cc:740] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:42.266366 25369 consensus_queue.cc:260] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:42.267637 25369 raft_consensus.cc:3060] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:42.270498 25158 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:42.273926 25245 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:42.275892 25352 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
W20251026 14:59:42.277127 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 341559us
I20251026 14:59:42.247035 25260 raft_consensus.cc:2749] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:42.222994 25197 consensus_queue.cc:1059] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20251026 14:59:42.246609 25210 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
W20251026 14:59:42.283023 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 229444us
I20251026 14:59:42.295423 22697 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:42.297152 22697 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:42.286723 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 229177us
W20251026 14:59:42.288244 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 214856us
I20251026 14:59:42.257186 25255 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
W20251026 14:59:42.325011 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 172149us
I20251026 14:59:42.329531 22773 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:42.283389 25226 raft_consensus.cc:3055] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:42.220037 25211 consensus_queue.cc:1059] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20251026 14:59:42.300135 25210 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:42.369113 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 114328us
I20251026 14:59:42.369906 22622 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:42.371232 22771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 196990us
I20251026 14:59:42.370002 22731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1152) took 1204 ms. Trace:
W20251026 14:59:42.372467 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:42.337488 25226 raft_consensus.cc:740] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:42.366415 22773 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:42.376070 22731 rpcz_store.cc:276] 1026 14:59:41.180104 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:41.704669 (+524565us) service_pool.cc:225] Handling call
1026 14:59:41.704948 (+ 279us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:41.705193 (+ 245us) raft_consensus.cc:1534] Early marking committed up to index 6
1026 14:59:42.028366 (+323173us) log.cc:844] Serialized 14 byte log entry
1026 14:59:42.368031 (+339665us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:42.368161 (+ 130us) raft_consensus.cc:1645] Marking committed up to 6
1026 14:59:42.368247 (+ 86us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:42.368385 (+ 138us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:42.369846 (+ 1461us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":479366,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":78476,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":14711,"thread_start_us":619617,"threads_started":3}
I20251026 14:59:42.229619 25343 consensus_queue.cc:260] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:42.367017 25260 raft_consensus.cc:2749] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251026 14:59:42.390394 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 93341.9us
I20251026 14:59:42.367853 25380 raft_consensus.cc:2804] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election won for term 8
W20251026 14:59:42.393244 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 88477.8us
I20251026 14:59:42.401988 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:59:42.401726 22697 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:42.372665 25211 consensus_queue.cc:1243] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Peer 562bb1e1c8e445b8baecd29c3669658e log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20251026 14:59:42.346439 25352 consensus_queue.cc:1048] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20251026 14:59:42.408553 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:42.411131 22624 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
W20251026 14:59:42.424365 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:42.373842 25192 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:42.385929 25158 raft_consensus.cc:2749] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:42.392298 25214 raft_consensus.cc:3055] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:42.400686 25287 raft_consensus.cc:2749] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:42.443579 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:42.456000 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 54388.4us
I20251026 14:59:42.457376 22697 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:42.424173 25384 raft_consensus.cc:2749] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:42.465073 22697 leader_election.cc:400] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 9. Current term is 10.
I20251026 14:59:42.467804 22697 leader_election.cc:403] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
W20251026 14:59:42.466184 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 54120.6us
I20251026 14:59:42.434152 25214 raft_consensus.cc:740] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:42.421875 24740 consensus_queue.cc:1048] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20251026 14:59:42.486552 22773 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:42.471688 25376 log.cc:927] Time spent T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
W20251026 14:59:42.473744 25374 log.cc:927] Time spent T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.129s user 0.000s sys 0.000s
I20251026 14:59:42.475605 25185 raft_consensus.cc:3060] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:42.519548 22773 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:42.476522 25253 raft_consensus.cc:2749] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251026 14:59:42.476748 25158 raft_consensus.cc:2749] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:42.442929 25226 consensus_queue.cc:260] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:42.477427 25193 raft_consensus.cc:3055] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:42.528218 25186 consensus_queue.cc:1048] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:59:42.522504 25194 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20251026 14:59:42.533947 25194 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:42.534448 25194 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Advancing to term 11
I20251026 14:59:42.544610 24740 consensus_queue.cc:1048] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20251026 14:59:42.482008 25373 log.cc:927] Time spent T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20251026 14:59:42.409883 25348 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:42.483584 22697 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:42.483999 22700 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:42.496188 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:42.527583 25264 raft_consensus.cc:2749] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:42.476388 25353 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:42.518298 25368 log.cc:927] Time spent T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.287s user 0.000s sys 0.001s
I20251026 14:59:42.487541 25192 raft_consensus.cc:2749] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 9. Current term is 10.
I20251026 14:59:42.583185 23404 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
W20251026 14:59:42.587384 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 103867us
I20251026 14:59:42.590126 22697 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:42.591965 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 180940us
I20251026 14:59:42.531325 25193 raft_consensus.cc:740] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:42.592962 22624 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:42.545593 25380 raft_consensus.cc:697] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:42.597077 22807 raft_consensus.cc:2393] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 8: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
W20251026 14:59:42.598461 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 114536us
I20251026 14:59:42.601938 22802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1364) took 1390 ms. Trace:
I20251026 14:59:42.606750 22802 rpcz_store.cc:276] 1026 14:59:41.211510 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:41.211896 (+ 386us) service_pool.cc:225] Handling call
1026 14:59:41.212179 (+ 283us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:41.300443 (+ 88264us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:42.601586 (+1301143us) spinlock_profiling.cc:243] Waited 87.3 ms on lock 0x7b540004e2d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:42.601853 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":200799488}
W20251026 14:59:42.609040 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:42.579443 25395 raft_consensus.cc:2804] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:42.519706 25384 consensus_queue.cc:1059] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.078s
W20251026 14:59:42.615269 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 245425us
I20251026 14:59:42.585261 25210 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:42.616571 22622 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:42.625414 25401 raft_consensus.cc:2749] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251026 14:59:42.594995 25402 raft_consensus.cc:2749] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:42.628983 25216 raft_consensus.cc:3055] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:42.628984 23399 raft_consensus.cc:1240] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []
W20251026 14:59:42.574196 25390 consensus_queue.cc:1175] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0d684445287b450a8f738b6ad4b59821" responder_term: 8 status { last_received { term: 6 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20251026 14:59:42.632025 22727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 1163) took 1447 ms. Trace:
I20251026 14:59:42.632228 25353 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:42.632473 22727 rpcz_store.cc:276] 1026 14:59:41.185103 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:41.305032 (+119929us) service_pool.cc:225] Handling call
1026 14:59:42.631800 (+1326768us) spinlock_profiling.cc:243] Waited 951 ms on lock 0x7b540040ddd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:42.631946 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2187929344}
I20251026 14:59:42.612751 25192 raft_consensus.cc:2804] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:42.629482 25216 raft_consensus.cc:740] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:42.671217 25192 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:42.671775 25192 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:42.671785 25216 consensus_queue.cc:260] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:42.651744 25358 log.cc:927] Time spent T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.674s user 0.004s sys 0.000s
I20251026 14:59:42.655984 24849 raft_consensus.cc:3055] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:42.674314 24849 raft_consensus.cc:740] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:42.660542 25241 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:42.675136 24849 consensus_queue.cc:260] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:42.676164 24849 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:42.676640 25241 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:42.600694 25214 consensus_queue.cc:260] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:42.618149 25395 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:42.677474 25395 raft_consensus.cc:3060] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
W20251026 14:59:42.677565 22732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 1172) took 1271 ms (client timeout 1607 ms). Trace:
I20251026 14:59:42.678495 25214 raft_consensus.cc:3060] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
W20251026 14:59:42.678165 22732 rpcz_store.cc:269] 1026 14:59:41.406136 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:41.416796 (+ 10660us) service_pool.cc:225] Handling call
1026 14:59:42.677352 (+1260556us) spinlock_profiling.cc:243] Waited 281 ms on lock 0x7b54000d66d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:42.677492 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":645254528}
I20251026 14:59:42.666642 22807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44594 (request call id 1384) took 1163 ms. Trace:
I20251026 14:59:42.679818 22807 rpcz_store.cc:276] 1026 14:59:41.502729 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:41.504514 (+ 1785us) service_pool.cc:225] Handling call
1026 14:59:42.666375 (+1161861us) spinlock_profiling.cc:243] Waited 870 ms on lock 0x7b540004b5d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:42.666560 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1999870720}
W20251026 14:59:42.635720 25236 consensus_meta.cc:287] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 1.241s user 0.016s sys 0.000s
I20251026 14:59:42.681366 25236 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:42.666915 25380 consensus_queue.cc:237] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:42.667832 25193 consensus_queue.cc:260] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:42.669890 25348 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:59:42.673189 25343 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:42.685911 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1145) took 1526 ms. Trace:
I20251026 14:59:42.689903 22726 rpcz_store.cc:276] 1026 14:59:41.159567 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:41.161799 (+ 2232us) service_pool.cc:225] Handling call
1026 14:59:41.162180 (+ 381us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:41.256826 (+ 94646us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:41.256885 (+ 59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:41.309110 (+ 52225us) log.cc:844] Serialized 26 byte log entry
1026 14:59:41.561034 (+251924us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:41.561098 (+ 64us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:41.561177 (+ 79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:42.650934 (+1089757us) spinlock_profiling.cc:243] Waited 3.94 ms on lock 0x7b440002304c. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8cdfc0905 00007fd8cdfc1322 00007fd8d0d0a231 00007fd8d0d05b5a 00007fd8d0cf991e 00007fd8d0cf8c66 00007fd8d0cf8888 00007fd8d0d022c1 00007fd8d0d02279 00007fd8d0d02209
1026 14:59:42.683011 (+ 32077us) raft_consensus.cc:1685] finished
1026 14:59:42.683036 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:42.685805 (+ 2769us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40112,"raft.queue_time_us":40701,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"spinlock_wait_cycles":9063808,"thread_start_us":261770,"threads_started":2,"wal-append.queue_time_us":553362}
I20251026 14:59:42.700275 22656 raft_consensus.cc:1240] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: []
W20251026 14:59:42.705237 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 94919.1us
I20251026 14:59:42.709090 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:42.709836 22802 raft_consensus.cc:1275] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20251026 14:59:42.710171 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:42.710561 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:42.711380 22806 raft_consensus.cc:2410] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20251026 14:59:42.710207 22802 pending_rounds.cc:85] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821: Aborting all ops after (but not including) 3
I20251026 14:59:42.712575 22802 pending_rounds.cc:107] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20251026 14:59:42.712785 22802 raft_consensus.cc:2889] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:42.713640 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:42.714836 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:42.715013 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:42.717037 22726 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 6.
I20251026 14:59:42.717540 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:42.718577 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:42.715400 22697 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:42.715550 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:42.720299 22808 raft_consensus.cc:2410] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 11 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20251026 14:59:42.716130 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:42.721460 22806 raft_consensus.cc:2393] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 8: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:42.722218 22697 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:42.723353 25412 raft_consensus.cc:2749] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:42.720521 22803 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:42.726612 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:42.733021 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:59:42.733098 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 121336us
I20251026 14:59:42.733911 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:42.734004 22700 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:42.734145 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:42.734529 22803 raft_consensus.cc:2468] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:42.734956 22800 raft_consensus.cc:2393] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 7: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:42.735033 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:42.735374 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:42.735711 22799 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
I20251026 14:59:42.736459 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:42.706683 25193 raft_consensus.cc:3060] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:42.742388 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:42.729244 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:42.729462 25406 consensus_queue.cc:1059] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20251026 14:59:42.698828 25285 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:42.731727 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:42.732368 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:42.745194 22807 raft_consensus.cc:2393] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 7: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:42.745548 22803 raft_consensus.cc:1240] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 8 LEADER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20251026 14:59:42.726758 22697 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:42.738317 25390 raft_consensus.cc:2749] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:42.739023 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:42.739550 25399 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:42.742739 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:42.698100 25273 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:42.749382 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:42.741019 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:42.697363 25287 consensus_queue.cc:1175] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "248493ec312e4398b828c6cb00e5c4b1" responder_term: 4 status { last_received { term: 3 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20251026 14:59:42.749336 22727 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 9.
I20251026 14:59:42.725818 25401 raft_consensus.cc:2749] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:42.744374 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:42.732590 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:42.745668 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:42.757200 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:42.758080 22730 raft_consensus.cc:2393] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 5: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:42.758788 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:42.759575 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:42.746486 22799 raft_consensus.cc:2393] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 7: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:42.746757 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:42.746984 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:42.747363 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:42.751051 22808 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
I20251026 14:59:42.742990 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:42.752122 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:42.752334 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:42.743393 22801 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 9.
I20251026 14:59:42.762228 22731 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:42.753494 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:42.763170 22806 raft_consensus.cc:2410] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251026 14:59:42.763084 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:42.765424 22807 raft_consensus.cc:1240] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20251026 14:59:42.766876 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:42.767505 22773 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:42.768584 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:42.728204 25241 consensus_queue.cc:1048] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20251026 14:59:42.769641 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:42.769887 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:42.772838 22729 raft_consensus.cc:1240] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251026 14:59:42.773797 22729 raft_consensus.cc:1240] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20251026 14:59:42.774075 22733 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:42.774750 22727 raft_consensus.cc:1240] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20251026 14:59:42.775279 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 48597.7us
I20251026 14:59:42.751504 25285 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:42.776219 22697 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:42.777258 22773 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:42.777845 25285 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:42.778695 25215 raft_consensus.cc:2804] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251026 14:59:42.779183 25215 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:42.779505 25215 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:42.778707 22773 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:42.782428 22804 raft_consensus.cc:1275] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20251026 14:59:42.783238 25185 raft_consensus.cc:2749] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:42.783329 22656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1517) took 1069 ms. Trace:
I20251026 14:59:42.784077 22656 rpcz_store.cc:276] 1026 14:59:41.714141 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:41.714271 (+ 130us) service_pool.cc:225] Handling call
1026 14:59:41.755362 (+ 41091us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:42.700877 (+945515us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:42.783074 (+ 82197us) spinlock_profiling.cc:243] Waited 945 ms on lock 0x7b54002a85d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:42.783249 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2172494208}
I20251026 14:59:42.785848 22734 raft_consensus.cc:2468] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:42.788381 25215 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:42.757438 25391 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:42.749222 25370 consensus_queue.cc:1048] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.043s
I20251026 14:59:42.791262 22773 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:42.754145 25399 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:42.956825 22804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1249) took 1156 ms. Trace:
W20251026 14:59:42.996176 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 220030us
I20251026 14:59:42.879370 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:42.985986 22730 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:42.997264 22697 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:42.984869 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 193648us
W20251026 14:59:42.959807 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 343296us
I20251026 14:59:42.948302 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:42.851684 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:42.975796 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:42.968273 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:42.879971 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:42.880537 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:43.003239 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:43.003597 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 410694us
I20251026 14:59:43.004212 22622 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:42.996097 22804 rpcz_store.cc:276] 1026 14:59:41.800474 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:41.909720 (+109246us) service_pool.cc:225] Handling call
1026 14:59:41.910333 (+ 613us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:42.782812 (+872479us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:42.956427 (+173615us) spinlock_profiling.cc:243] Waited 872 ms on lock 0x7b540004ced4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:42.956729 (+ 302us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2004746496}
I20251026 14:59:43.004997 22652 raft_consensus.cc:2393] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 7: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:43.002763 22650 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:42.998409 22651 raft_consensus.cc:2393] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 6: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
W20251026 14:59:43.007750 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:43.008668 22773 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:43.008585 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:43.009089 22622 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:43.015100 25185 raft_consensus.cc:2804] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:43.015619 25185 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:43.016064 25185 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:43.016373 22622 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:43.016745 22773 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:43.018407 25404 raft_consensus.cc:2749] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251026 14:59:43.019093 25214 raft_consensus.cc:2749] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:43.019377 25404 raft_consensus.cc:2804] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:43.020217 25404 raft_consensus.cc:697] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:42.965021 22805 raft_consensus.cc:2468] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:43.021565 25404 consensus_queue.cc:237] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.021121 22773 leader_election.cc:304] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:43.024278 22622 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:43.003261 22657 raft_consensus.cc:2468] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
W20251026 14:59:42.862752 25281 consensus_meta.cc:287] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.603s user 0.012s sys 0.000s
I20251026 14:59:42.791747 25389 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:42.844586 25273 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:42.866588 25194 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:42.947803 25307 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:43.012315 22624 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:42.778654 25226 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:42.868510 25352 raft_consensus.cc:3055] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:42.987897 25216 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:43.010586 25397 raft_consensus.cc:2804] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:42.789232 25369 raft_consensus.cc:2749] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:43.030138 22622 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:42.876000 25366 log.cc:927] Time spent T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.626s user 0.003s sys 0.003s
I20251026 14:59:43.032172 22773 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:43.024518 25193 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:42.793804 25215 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:42.947814 25408 raft_consensus.cc:2749] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:43.005993 25398 raft_consensus.cc:2749] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:43.086684 22650 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 9.
I20251026 14:59:42.826143 25370 raft_consensus.cc:2749] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20251026 14:59:42.960089 25400 raft_consensus.cc:2804] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:43.092000 25400 raft_consensus.cc:697] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:42.830792 25411 raft_consensus.cc:2749] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251026 14:59:43.094583 25194 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 11 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:43.059167 22732 raft_consensus.cc:1275] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20251026 14:59:43.095829 22732 pending_rounds.cc:85] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: Aborting all ops after (but not including) 7
I20251026 14:59:43.096127 22732 pending_rounds.cc:107] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20251026 14:59:43.096371 22732 raft_consensus.cc:2889] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:43.095506 25400 consensus_queue.cc:237] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.049093 25287 consensus_queue.cc:1059] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20251026 14:59:43.102167 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:43.104939 22728 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:43.107903 22732 raft_consensus.cc:1240] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20251026 14:59:43.082484 25397 raft_consensus.cc:697] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:43.057385 25307 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:43.111719 22659 raft_consensus.cc:1240] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:43.071861 22622 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:43.072690 25233 raft_consensus.cc:2804] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:43.027546 25214 raft_consensus.cc:2749] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251026 14:59:43.073778 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 76581.1us
I20251026 14:59:43.116271 22697 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:42.864616 25192 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:43.076733 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 44669.3us
I20251026 14:59:43.044585 25186 consensus_queue.cc:1059] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20251026 14:59:43.077247 25196 raft_consensus.cc:2804] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:42.791684 25236 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:43.042941 25124 raft_consensus.cc:2804] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251026 14:59:43.045784 22553 catalog_manager.cc:5649] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 2 to 4, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 4 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:42.793812 25255 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.056052 25380 raft_consensus.cc:2749] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:43.047574 25399 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:42.875073 25379 raft_consensus.cc:2804] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:43.068099 25285 raft_consensus.cc:2804] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:43.061998 25352 raft_consensus.cc:740] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:42.789784 25391 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.078614 25389 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.080431 25403 consensus_queue.cc:1048] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20251026 14:59:43.101845 25193 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:43.285260 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 169058us
I20251026 14:59:43.169133 22728 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 10.
I20251026 14:59:43.286273 22697 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:43.172289 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:43.152688 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:43.174418 22725 raft_consensus.cc:2393] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 9: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
W20251026 14:59:43.287853 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 229980us
I20251026 14:59:43.289126 25352 consensus_queue.cc:260] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.152947 25384 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:43.291990 25384 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.287658 22659 raft_consensus.cc:2393] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 7: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:43.127884 22622 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:43.284201 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 164318us
W20251026 14:59:43.296512 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 168712us
I20251026 14:59:43.297247 22622 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:43.298388 22624 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:43.299422 22700 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:43.223242 25404 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:43.307518 25307 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:43.283809 24740 raft_consensus.cc:3055] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:43.127303 25196 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:43.124738 25398 raft_consensus.cc:2804] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:43.288206 25285 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:43.319490 22651 raft_consensus.cc:1240] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:43.323132 22697 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:43.315308 22624 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:43.189841 25348 raft_consensus.cc:3055] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:43.120805 25041 raft_consensus.cc:2804] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:43.312532 22553 catalog_manager.cc:5649] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 5 to 6, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 6 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:59:43.227944 25281 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:43.326965 25399 raft_consensus.cc:2749] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:43.327392 25410 raft_consensus.cc:2749] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251026 14:59:43.326781 25404 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:43.328408 25408 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:43.330632 25389 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:43.257771 25416 log.cc:927] Time spent T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.131s user 0.000s sys 0.001s
I20251026 14:59:43.293761 25379 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:43.124342 25233 raft_consensus.cc:697] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:43.335980 22622 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:43.296067 25124 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:43.336266 25411 raft_consensus.cc:493] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:43.309360 25273 raft_consensus.cc:2804] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:43.314491 25403 raft_consensus.cc:2804] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:43.351398 22697 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:43.360584 25041 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:43.324277 25236 raft_consensus.cc:3060] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:43.342963 22773 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:43.276082 25185 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.333514 24740 raft_consensus.cc:740] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:43.359452 22553 catalog_manager.cc:5649] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 3 to 6, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 6 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:59:43.385545 22553 catalog_manager.cc:5649] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 4 to 5, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 5 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:43.336649 25186 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:43.341150 25196 raft_consensus.cc:3060] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:43.349179 25398 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:43.269006 25397 consensus_queue.cc:237] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.347976 25192 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:43.351491 25285 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:43.352303 25353 raft_consensus.cc:2749] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251026 14:59:43.368947 25281 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:43.368541 25415 log.cc:927] Time spent T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20251026 14:59:43.349063 25380 consensus_queue.cc:1048] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20251026 14:59:43.356688 25348 raft_consensus.cc:740] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:43.381089 22697 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:43.323808 22700 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:43.395274 25389 raft_consensus.cc:2804] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader election won for term 10
W20251026 14:59:43.437052 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 56020.5us
I20251026 14:59:43.392897 25391 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:43.378123 25124 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:43.397111 22553 catalog_manager.cc:5649] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:43.403818 25193 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:43.387384 25041 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:43.381556 25384 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:43.333873 25214 consensus_queue.cc:1059] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20251026 14:59:43.370636 25408 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.456413 25380 raft_consensus.cc:2804] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader election won for term 9
I20251026 14:59:43.451632 25410 raft_consensus.cc:2749] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:43.456931 25408 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:43.457926 25408 raft_consensus.cc:2804] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:43.458526 25408 raft_consensus.cc:697] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:43.458676 25404 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:43.458194 22553 catalog_manager.cc:5649] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 3 to 6, leader changed from df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 6 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
W20251026 14:59:43.429369 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 125542us
I20251026 14:59:43.372983 25379 raft_consensus.cc:3060] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
W20251026 14:59:43.437846 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 114048us
I20251026 14:59:43.399084 25186 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.449631 22697 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:43.451184 22774 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.2.1:46185 timed out after 1.634s (SENT) [suppressed 27 similar messages]
W20251026 14:59:43.459676 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 123794us
I20251026 14:59:43.460227 22802 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
W20251026 14:59:43.448359 25343 consensus_meta.cc:287] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.752s user 0.009s sys 0.003s
I20251026 14:59:43.402390 25398 raft_consensus.cc:3060] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:43.441148 25192 raft_consensus.cc:2749] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:43.464270 22700 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:43.382705 25411 raft_consensus.cc:515] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:43.476751 22697 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:43.468999 22726 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:43.437062 25400 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:43.467244 22622 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:43.485257 25404 raft_consensus.cc:2804] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:43.486039 25404 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:43.486537 25404 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:43.461472 25389 raft_consensus.cc:697] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 10 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:43.495527 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:43.473883 22553 catalog_manager.cc:5649] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 3 to 4, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 4 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
W20251026 14:59:43.470948 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 155726us
I20251026 14:59:43.465225 25255 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:43.494874 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:43.456959 25380 raft_consensus.cc:697] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:43.511222 22727 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Advancing to term 11
I20251026 14:59:43.511581 25403 raft_consensus.cc:697] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:43.495793 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:43.495086 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:43.520931 22730 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:43.521031 22655 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:43.490643 25348 consensus_queue.cc:260] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.512151 22622 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:43.515065 22732 pending_rounds.cc:85] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e: Aborting all ops after (but not including) 4
W20251026 14:59:43.515486 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 64338.2us
I20251026 14:59:43.527818 22732 pending_rounds.cc:107] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20251026 14:59:43.528121 22732 raft_consensus.cc:2889] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:43.512224 22624 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:43.466784 25273 raft_consensus.cc:697] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:43.519081 22697 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:43.477421 25391 raft_consensus.cc:3055] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 8 LEADER]: Stepping down as leader of term 8
I20251026 14:59:43.442476 25233 consensus_queue.cc:237] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.481863 25399 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:43.522961 22700 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:43.523561 22553 catalog_manager.cc:5649] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
W20251026 14:59:43.543614 25419 log.cc:927] Time spent T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.155s user 0.000s sys 0.001s
I20251026 14:59:43.543385 22553 catalog_manager.cc:5649] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 2 to 4, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 4 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:59:43.490048 24849 raft_consensus.cc:2804] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader election won for term 9
I20251026 14:59:43.526971 22773 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:43.522903 25402 raft_consensus.cc:2764] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20251026 14:59:43.491621 25343 raft_consensus.cc:2804] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:43.535225 22697 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:43.514134 25353 raft_consensus.cc:2804] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:43.445173 24740 consensus_queue.cc:260] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.468145 25401 consensus_queue.cc:1059] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.054s
W20251026 14:59:43.590668 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 78511.9us
I20251026 14:59:43.551169 22803 raft_consensus.cc:1240] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 11 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20251026 14:59:43.518481 25185 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:43.551914 25253 raft_consensus.cc:2804] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election won for term 8
W20251026 14:59:43.552067 25422 log.cc:927] Time spent T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.153s user 0.001s sys 0.000s
W20251026 14:59:43.552685 25424 log.cc:927] Time spent T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
I20251026 14:59:43.525466 25354 consensus_queue.cc:1059] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20251026 14:59:43.553527 25436 consensus_queue.cc:1048] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251026 14:59:43.556735 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:43.557369 25411 leader_election.cc:290] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:43.521992 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:43.566653 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:43.569257 22655 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5"
dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:46598
I20251026 14:59:43.631031 22655 raft_consensus.cc:606] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:43.631450 22655 raft_consensus.cc:614] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251026 14:59:43.570785 22654 raft_consensus.cc:1240] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251026 14:59:43.571161 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:43.572301 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:43.579228 22651 raft_consensus.cc:1240] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20251026 14:59:43.580487 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 45369us
I20251026 14:59:43.636214 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:43.637352 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:43.580317 25216 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:43.464015 25408 consensus_queue.cc:237] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.561735 24849 raft_consensus.cc:697] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [term 9 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:43.568333 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:43.653486 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:43.545166 22553 catalog_manager.cc:5649] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 4 to 5, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 5 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:43.591574 22624 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:43.515241 25400 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.593256 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:43.593740 22653 raft_consensus.cc:1275] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20251026 14:59:43.658006 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 66507.6us
I20251026 14:59:43.659178 22624 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:43.596817 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:43.597385 25435 raft_consensus.cc:2749] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251026 14:59:43.597471 25389 consensus_queue.cc:237] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.598311 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:43.673385 22803 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:43.598302 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:43.550891 25348 raft_consensus.cc:3060] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:43.601576 25380 consensus_queue.cc:237] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.537012 25391 raft_consensus.cc:740] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:43.602531 25273 consensus_queue.cc:237] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.538511 25384 raft_consensus.cc:2749] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:43.618835 22799 raft_consensus.cc:1240] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
W20251026 14:59:43.572127 25395 consensus_meta.cc:287] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.894s user 0.001s sys 0.012s
W20251026 14:59:43.552726 25425 log.cc:927] Time spent T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.143s user 0.000s sys 0.001s
I20251026 14:59:43.715946 22803 raft_consensus.cc:2468] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:43.722257 22653 raft_consensus.cc:1240] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:43.722641 22803 raft_consensus.cc:1240] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20251026 14:59:43.588584 22727 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 11.
I20251026 14:59:43.590023 25343 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:43.634554 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:43.732565 22651 raft_consensus.cc:2468] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:43.681186 25406 raft_consensus.cc:493] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:43.592571 25214 raft_consensus.cc:2804] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:43.603977 25212 raft_consensus.cc:2804] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader pre-election won for term 4
W20251026 14:59:43.545125 25420 log.cc:927] Time spent T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20251026 14:59:43.616659 25253 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:43.662878 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:43.664431 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:43.665988 22659 raft_consensus.cc:1240] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 8. Current term is 9. Ops: [7.4-8.5]
I20251026 14:59:43.666036 23404 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:43.667099 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:43.771535 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:43.562670 25399 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:43.660349 22734 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:43.545703 25199 consensus_queue.cc:1048] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.098s
I20251026 14:59:43.623793 25215 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:43.676321 22697 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:43.793797 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 117598us
I20251026 14:59:43.643676 25255 raft_consensus.cc:2804] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:43.635476 25401 raft_consensus.cc:3055] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:43.704653 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:43.652549 24740 raft_consensus.cc:3060] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:43.708098 22658 raft_consensus.cc:2393] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 8: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:43.625916 25192 consensus_queue.cc:1059] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20251026 14:59:43.620757 25369 consensus_queue.cc:1059] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20251026 14:59:43.652741 25236 raft_consensus.cc:2749] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:43.672565 25186 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:43.801872 24849 consensus_queue.cc:237] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.831669 25186 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:43.832134 25186 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.613183 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:43.787968 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:43.679948 25281 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:43.650882 25196 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.744287 22800 raft_consensus.cc:1240] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:43.744936 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:43.655779 25216 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.783969 22806 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:43.853513 25186 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:43.855516 25379 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.798096 25185 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:43.792264 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:43.782148 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 119938us
I20251026 14:59:43.858657 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:35078 (request call id 176) took 1091 ms. Trace:
I20251026 14:59:43.859077 22725 rpcz_store.cc:276] 1026 14:59:42.767052 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:43.320939 (+553887us) service_pool.cc:225] Handling call
1026 14:59:43.858149 (+537210us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b5400076554. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c7fe0 00007fd8d1166aca 00007fd8d610b01e 00007fd8cf6b7437 00007fd8cf6b73c2 00007fd8cf6b72f1 00007fd8cf6b726c 00007fd8cf6b6522 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:43.858583 (+ 434us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":293442304}
I20251026 14:59:43.741916 25343 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:43.731209 25405 raft_consensus.cc:2804] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader election won for term 9
W20251026 14:59:43.782235 25421 log.cc:927] Time spent T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20251026 14:59:43.722548 25095 consensus_queue.cc:1048] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.067s
W20251026 14:59:43.773738 25226 consensus_meta.cc:287] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.717s user 0.013s sys 0.000s
I20251026 14:59:43.651062 25403 consensus_queue.cc:237] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.747473 25214 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:43.797557 25425 log.cc:927] Time spent T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.065s user 0.000s sys 0.001s
I20251026 14:59:43.757802 25406 raft_consensus.cc:515] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:43.823385 25442 raft_consensus.cc:2804] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:43.824071 25384 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:43.825801 25440 log.cc:927] Time spent T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20251026 14:59:43.826941 25404 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.828210 25399 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:59:43.829645 25444 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:43.802059 25391 consensus_queue.cc:260] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.788235 25253 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:43.758147 25212 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:43.802183 25401 raft_consensus.cc:740] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:43.790131 25438 raft_consensus.cc:2804] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:43.807790 25255 raft_consensus.cc:697] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:43.724807 25395 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.803818 25215 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.849850 25369 raft_consensus.cc:3055] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:43.857604 25185 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.872857 25214 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:43.871618 25405 raft_consensus.cc:697] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:43.905781 25400 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:43.900739 25444 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.926095 25406 leader_election.cc:290] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:43.880815 25442 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:43.907236 25352 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:43.926769 25442 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:43.923560 25405 consensus_queue.cc:237] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.931043 25285 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.926184 25444 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:43.933596 25255 consensus_queue.cc:237] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.941455 25353 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:43.942190 25353 raft_consensus.cc:3060] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:43.918201 25369 raft_consensus.cc:740] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:43.950596 25411 raft_consensus.cc:3055] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:43.951021 25411 raft_consensus.cc:740] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:43.952062 25395 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:43.951810 25411 consensus_queue.cc:260] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.910113 25438 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:43.956588 25438 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:43.884428 25384 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.913681 25379 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:43.922763 25391 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:43.959233 25401 consensus_queue.cc:260] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:43.680814 22553 catalog_manager.cc:5649] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 3 to 5, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 5 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:44.025977 22553 catalog_manager.cc:5649] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 2 to 4, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 4 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:43.804121 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:44.041260 22654 raft_consensus.cc:1240] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20251026 14:59:44.047873 22657 raft_consensus.cc:1240] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:44.041340 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:44.055994 22657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1314) took 1005 ms. Trace:
I20251026 14:59:44.056936 22697 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:44.056838 22657 rpcz_store.cc:276] 1026 14:59:43.050247 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:43.051877 (+ 1630us) service_pool.cc:225] Handling call
1026 14:59:43.052290 (+ 413us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:44.050352 (+998062us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:44.055471 (+ 5119us) spinlock_profiling.cc:243] Waited 995 ms on lock 0x7b54001fc054. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:44.055866 (+ 395us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2288151424}
I20251026 14:59:44.074962 22728 raft_consensus.cc:2393] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 8: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:44.075083 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 1525) took 1327 ms. Trace:
I20251026 14:59:43.830235 22729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1540) took 1075 ms. Trace:
I20251026 14:59:44.075727 22726 rpcz_store.cc:276] 1026 14:59:42.747304 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:42.751884 (+ 4580us) service_pool.cc:225] Handling call
1026 14:59:44.074502 (+1322618us) spinlock_profiling.cc:243] Waited 706 ms on lock 0x7b540028b754. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:44.074768 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1622718208}
I20251026 14:59:44.076408 22729 rpcz_store.cc:276] 1026 14:59:42.755151 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:42.826661 (+ 71510us) service_pool.cc:225] Handling call
1026 14:59:42.999176 (+172515us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:43.062372 (+ 63196us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:43.062407 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:43.062991 (+ 584us) log.cc:844] Serialized 26 byte log entry
1026 14:59:43.102627 (+ 39636us) spinlock_profiling.cc:243] Waited 5.48 ms on lock 0x7b180034f840. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce0942f7 00007fd8d6e6d219 00007fd8ce0b0c4b 00007fd8ce0aeeef 00007fd8ce0b11b2 00007fd8d0cf82bb 00007fd8d0cfddc3 00007fd8d0cfe171 00007fd8d111965f
1026 14:59:43.378142 (+275515us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:43.378185 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:43.378259 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:43.666978 (+288719us) raft_consensus.cc:1685] finished
1026 14:59:43.667001 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:43.830053 (+163052us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b54000b86d0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:43.830171 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":76091,"raft.queue_time_us":62862,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":9906,"spinlock_wait_cycles":408301696,"thread_start_us":275322,"threads_started":1,"wal-append.queue_time_us":322310}
I20251026 14:59:43.858672 22730 raft_consensus.cc:2468] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 5.
W20251026 14:59:44.082391 22732 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.010s sys 0.006s
I20251026 14:59:44.082690 22731 raft_consensus.cc:2410] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20251026 14:59:44.083061 22732 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.554s user 0.010s sys 0.006s
I20251026 14:59:44.092195 22731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 1530) took 1343 ms. Trace:
I20251026 14:59:44.096678 22731 rpcz_store.cc:276] 1026 14:59:42.748599 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:42.762838 (+ 14239us) service_pool.cc:225] Handling call
1026 14:59:44.091839 (+1329001us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b540030bbd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:44.092101 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3030460672}
W20251026 14:59:44.110263 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:44.112309 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:43.867161 22806 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 9.
I20251026 14:59:43.870627 22808 raft_consensus.cc:1240] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:44.148461 25193 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:44.149927 25193 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.155236 25194 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:43.908041 25216 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:44.170137 25370 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:43.953060 25411 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:44.029808 22553 catalog_manager.cc:5649] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
W20251026 14:59:44.191104 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:43.899812 25212 raft_consensus.cc:3060] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:44.156409 25194 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.193827 25344 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
W20251026 14:59:44.191208 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 134609us
W20251026 14:59:44.192589 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 532195us
I20251026 14:59:44.194919 25379 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
W20251026 14:59:44.216410 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:44.216212 25379 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:44.201946 25401 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:44.219080 25369 consensus_queue.cc:260] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:44.221122 25216 consensus_queue.cc:1175] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "562bb1e1c8e445b8baecd29c3669658e" responder_term: 6 status { last_received { term: 6 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 6 index: 4 } }
I20251026 14:59:44.202069 25287 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:44.224047 25287 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.205039 25395 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:44.226337 25395 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:44.206167 25389 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:44.207841 25196 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:44.205521 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:44.230614 25404 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:44.195178 25370 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.202432 25285 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:44.229135 25389 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.213893 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:44.209407 22697 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:44.209008 25185 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:44.236722 25285 raft_consensus.cc:2749] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:44.237174 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:44.246865 25287 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:44.248402 25307 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:44.270802 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 61624us
I20251026 14:59:44.249221 25307 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:44.281991 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:44.268121 25193 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:44.285163 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:44.287142 25186 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
W20251026 14:59:44.287732 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 76559.6us
W20251026 14:59:44.288764 22650 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.010s sys 0.022s
I20251026 14:59:44.288264 25186 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.288844 22624 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:44.291329 25389 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:44.291956 22773 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:44.298126 25370 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:44.301486 25395 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
W20251026 14:59:44.303012 22604 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6a1f6f52f31d4a9e98b2c86f2358ea8b, TS df3a0d1619ee48e8ab68ee0543a72a29 -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:44.306051 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:44.323614 22650 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s user 0.011s sys 0.022s
I20251026 14:59:44.323730 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:44.341511 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 52887.7us
I20251026 14:59:44.331097 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:44.329635 22771 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.009s sys 0.016s
W20251026 14:59:44.350417 22604 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 6a1f6f52f31d4a9e98b2c86f2358ea8b, TS df3a0d1619ee48e8ab68ee0543a72a29 -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.332149 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:44.342674 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 65822.8us
W20251026 14:59:44.364850 22771 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.751s user 0.009s sys 0.017s
I20251026 14:59:44.336555 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:44.341914 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:44.325614 24740 consensus_meta.cc:287] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.521s user 0.010s sys 0.011s
I20251026 14:59:44.334101 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:44.319836 25307 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:44.317961 25379 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
W20251026 14:59:44.408124 25466 log.cc:927] Time spent T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
W20251026 14:59:44.408306 25343 consensus_meta.cc:287] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.548s user 0.007s sys 0.007s
I20251026 14:59:44.317183 25442 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.378923 25193 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:44.379683 22624 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:44.380404 25436 raft_consensus.cc:2804] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:44.323390 25215 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:44.382472 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:44.387439 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:44.391772 25185 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:44.306111 22553 catalog_manager.cc:5649] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
W20251026 14:59:44.400941 25398 consensus_meta.cc:287] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.910s user 0.010s sys 0.005s
I20251026 14:59:44.401724 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:44.402160 25438 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.422145 22659 raft_consensus.cc:1240] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:44.423297 25438 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:44.424098 25389 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 11 FOLLOWER]: Leader election won for term 11
I20251026 14:59:44.424854 25389 raft_consensus.cc:697] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [term 11 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:44.425529 22653 raft_consensus.cc:2468] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:44.426172 25389 consensus_queue.cc:237] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.427235 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:44.428434 22653 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
I20251026 14:59:44.320555 25369 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:44.429363 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:44.431102 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:44.431918 22655 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:44.432067 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:44.432832 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:44.432533 22553 catalog_manager.cc:5649] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 1 to 4, leader changed from df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 4 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:44.433256 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:44.434868 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:44.434624 22553 catalog_manager.cc:5649] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 5 to 7, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 7 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:44.435732 22659 raft_consensus.cc:2410] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251026 14:59:44.322427 25194 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:44.437415 22658 raft_consensus.cc:2410] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20251026 14:59:44.439582 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:44.328440 25186 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:44.338127 25384 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:44.329540 25287 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
W20251026 14:59:44.381374 25041 consensus_meta.cc:287] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.926s user 0.009s sys 0.008s
I20251026 14:59:44.413730 22656 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
W20251026 14:59:44.410717 25472 log.cc:927] Time spent T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20251026 14:59:44.408813 25343 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.415215 22808 raft_consensus.cc:2468] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 4.
I20251026 14:59:44.420912 22652 raft_consensus.cc:1240] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:44.439069 25185 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.453256 25287 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.453913 25285 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:44.454443 25410 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:44.454917 25199 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:44.413457 25436 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:44.483541 25226 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:44.484288 25226 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.436496 22553 catalog_manager.cc:5649] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 1 to 2, leader changed from df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 2 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
W20251026 14:59:44.450624 22773 leader_election.cc:336] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.783s (SENT)
I20251026 14:59:44.484418 25193 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:44.566187 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 294163us
I20251026 14:59:44.566933 22658 raft_consensus.cc:1240] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: []
W20251026 14:59:44.526584 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:44.572500 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 196400us
I20251026 14:59:44.572903 25226 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:44.484305 25470 log.cc:927] Time spent T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20251026 14:59:44.575320 25344 raft_consensus.cc:2749] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:44.575685 25442 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:44.450358 25398 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.576848 25185 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:44.525439 25395 raft_consensus.cc:2804] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election won for term 8
W20251026 14:59:44.587661 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:44.574193 22624 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:44.552188 25348 consensus_meta.cc:287] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.686s user 0.014s sys 0.000s
I20251026 14:59:44.482898 25214 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.485890 25436 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:44.579201 25458 consensus_queue.cc:1059] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20251026 14:59:44.594328 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:44.597972 25124 consensus_meta.cc:287] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 1.135s user 0.013s sys 0.004s
I20251026 14:59:44.598580 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:44.579824 25285 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.600466 25212 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:44.470518 25041 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.554491 25410 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:44.600692 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 180377us
I20251026 14:59:44.609634 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1319) took 1325 ms. Trace:
W20251026 14:59:44.602216 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 92087us
I20251026 14:59:44.553941 24849 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:44.554728 25199 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.554170 25444 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:44.626068 22652 rpcz_store.cc:276] 1026 14:59:43.284781 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:43.285017 (+ 236us) service_pool.cc:225] Handling call
1026 14:59:43.319323 (+ 34306us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:44.478880 (+1159557us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:44.609333 (+130453us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54001d9cd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:44.609547 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2522950528}
I20251026 14:59:44.625021 25395 raft_consensus.cc:697] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:44.621979 22553 catalog_manager.cc:5649] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 2 to 4, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 4 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:44.634193 22655 raft_consensus.cc:2468] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:44.640820 22651 raft_consensus.cc:1240] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:44.641453 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:44.634241 23367 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters:
W20251026 14:59:44.646500 25474 log.cc:927] Time spent T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.098s user 0.000s sys 0.001s
I20251026 14:59:44.646746 22697 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:44.626746 25193 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:44.651480 25458 raft_consensus.cc:2804] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader election won for term 9
I20251026 14:59:44.652199 25287 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:44.652361 25458 raft_consensus.cc:697] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 9 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:44.659638 25041 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:44.658991 22553 catalog_manager.cc:5649] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 1 to 4, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 4 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:44.661499 25212 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Requested vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:44.662045 22773 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:44.662395 25343 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:44.662848 25199 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:44.664494 25253 consensus_meta.cc:287] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.745s user 0.005s sys 0.009s
I20251026 14:59:44.634652 25438 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:44.668097 22553 catalog_manager.cc:5649] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:44.664311 22773 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:44.627110 25124 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.664422 25391 raft_consensus.cc:2804] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:44.684689 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:44.687070 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 113073us
W20251026 14:59:44.688197 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 173777us
I20251026 14:59:44.665083 25253 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.690770 25369 raft_consensus.cc:2749] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:44.690891 25398 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:44.687033 25343 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:44.689553 22657 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 10.
I20251026 14:59:44.677830 25391 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:44.686192 22553 catalog_manager.cc:5649] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:44.692528 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:44.666008 25444 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.700397 25241 raft_consensus.cc:3055] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:44.697518 25429 raft_consensus.cc:2804] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:44.676440 24849 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.697021 25391 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:44.706800 25429 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:44.708415 22553 catalog_manager.cc:5649] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 2 to 4, leader changed from df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 4 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } }
I20251026 14:59:44.703699 25241 raft_consensus.cc:740] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:44.712767 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:44.719838 22624 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:44.720834 22700 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:44.707576 25124 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:44.696552 25343 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.709539 25214 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:44.710767 25429 raft_consensus.cc:3060] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:44.729808 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:44.696429 25458 consensus_queue.cc:237] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.714736 25285 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:44.730855 22727 raft_consensus.cc:1240] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:44.730582 22553 catalog_manager.cc:5649] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 2 to 5, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 5 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:44.731300 25410 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:44.741730 22553 catalog_manager.cc:5649] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 4 to 5, leader changed from df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 5 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } }
I20251026 14:59:44.743702 22624 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:44.744707 25124 raft_consensus.cc:2804] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:44.745258 25124 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:44.745744 25124 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:44.747984 23369 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:44.749676 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:44.751705 25343 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:44.727176 25287 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:44.752385 25193 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:44.753175 25193 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.753638 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:44.754407 22807 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
I20251026 14:59:44.755640 25193 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:44.755868 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:44.756798 22800 raft_consensus.cc:2410] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251026 14:59:44.736075 25401 raft_consensus.cc:3055] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:44.739128 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:44.740317 22658 raft_consensus.cc:3055] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 LEADER]: Stepping down as leader of term 9
I20251026 14:59:44.740988 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:44.752465 25196 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:44.757241 25450 raft_consensus.cc:2804] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:44.761385 25450 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:44.761958 25450 raft_consensus.cc:3060] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:44.762905 22624 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:44.741771 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:44.758163 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:44.758322 25401 raft_consensus.cc:740] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:44.767263 25401 consensus_queue.cc:260] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.768558 25401 raft_consensus.cc:3060] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:44.758730 22651 raft_consensus.cc:2393] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 7: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:44.758863 22658 raft_consensus.cc:740] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:44.770931 25041 raft_consensus.cc:2749] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:44.769317 22553 catalog_manager.cc:5649] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:44.771449 22658 consensus_queue.cc:260] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.772588 22658 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:44.759630 22652 raft_consensus.cc:2393] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 8: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:44.759938 25196 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.767252 25405 raft_consensus.cc:3060] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:44.776124 25196 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:44.758857 25444 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:44.776706 25253 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:44.733675 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:44.778193 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 134199us
W20251026 14:59:44.792520 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:44.793480 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:44.803926 25479 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:44.804322 22804 raft_consensus.cc:2393] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 8: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:44.804664 25479 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.807677 25241 consensus_queue.cc:260] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:44.810098 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:44.709335 25395 consensus_queue.cc:237] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.815596 22624 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:44.822659 22807 raft_consensus.cc:1240] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20251026 14:59:44.752529 25287 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.800768 25398 raft_consensus.cc:2804] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election won for term 9
W20251026 14:59:44.841863 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:44.840553 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:44.793987 25343 consensus_queue.cc:1059] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20251026 14:59:44.835175 25241 raft_consensus.cc:3060] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:44.834116 25041 raft_consensus.cc:2749] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:44.840075 25398 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:44.852707 25253 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:44.872617 22727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1558) took 1305 ms. Trace:
I20251026 14:59:44.865788 25398 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:44.881815 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:44.883073 25479 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:44.884549 25390 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:44.878419 22727 rpcz_store.cc:276] 1026 14:59:43.564392 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:43.728240 (+163848us) service_pool.cc:225] Handling call
1026 14:59:43.728648 (+ 408us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:44.751369 (+1022721us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:44.855484 (+104115us) spinlock_profiling.cc:243] Waited 978 ms on lock 0x7b54000b9ad4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:44.869216 (+ 13732us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2248643968}
I20251026 14:59:44.900900 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:44.906770 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:44.907824 22731 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 9.
I20251026 14:59:44.908524 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:44.909183 22807 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
I20251026 14:59:44.906803 25444 raft_consensus.cc:3055] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:44.910422 25444 raft_consensus.cc:740] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:44.911310 25444 consensus_queue.cc:260] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.912515 25444 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:44.914467 22804 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:44.937741 25041 consensus_queue.cc:1048] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20251026 14:59:44.945508 25212 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:44.946483 25212 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:44.949697 25353 consensus_meta.cc:287] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.956s user 0.017s sys 0.000s
I20251026 14:59:44.950340 25353 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.954265 25287 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:44.962242 25438 raft_consensus.cc:2749] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:44.950197 22624 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:44.978057 22654 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
W20251026 14:59:44.978884 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 78252.9us
I20251026 14:59:44.900286 24849 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:44.963486 25485 log.cc:927] Time spent T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20251026 14:59:44.889817 25199 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:44.982204 25199 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.984618 22804 raft_consensus.cc:1240] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 10 LEADER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20251026 14:59:44.965214 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:44.985833 22733 raft_consensus.cc:2468] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 4.
I20251026 14:59:44.987717 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:44.988708 22799 raft_consensus.cc:1275] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:44.985107 25199 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:44.989609 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:44.990265 22800 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:44.990896 22803 raft_consensus.cc:1275] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251026 14:59:44.992975 25429 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:44.993790 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:44.994653 23403 raft_consensus.cc:2368] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5.
I20251026 14:59:44.995671 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:44.998339 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:44.890318 25253 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.908993 25481 consensus_queue.cc:1059] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.120s
I20251026 14:59:44.967344 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.002031 22732 raft_consensus.cc:2468] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:45.002378 25481 raft_consensus.cc:3055] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:45.002874 25481 raft_consensus.cc:740] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:45.004130 25481 consensus_queue.cc:260] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:44.901597 25343 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:45.005566 25343 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.008533 25343 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:44.969849 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:44.989802 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:45.010180 22726 raft_consensus.cc:3060] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:44.968624 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.010797 22729 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
W20251026 14:59:45.004863 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:45.011866 25405 raft_consensus.cc:2749] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:45.012826 25405 raft_consensus.cc:2749] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251026 14:59:44.927335 25390 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.014648 25481 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:45.015612 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:44.965358 22697 leader_election.cc:304] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:45.018108 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.018468 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.050814 22732 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 9.
I20251026 14:59:45.018769 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.019308 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.020282 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.020321 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:44.953025 25353 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:45.001205 25253 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:45.026099 25343 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:44.981033 22624 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:45.034909 25124 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.036263 22658 raft_consensus.cc:1275] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20251026 14:59:45.060539 22729 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 10.
I20251026 14:59:45.060763 22658 pending_rounds.cc:85] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29: Aborting all ops after (but not including) 5
I20251026 14:59:45.052181 22730 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:45.063177 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.063001 22725 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:45.063400 25343 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.069586 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.070253 22727 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:45.070827 25343 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:45.071184 22734 raft_consensus.cc:2410] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251026 14:59:45.062269 22658 pending_rounds.cc:107] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20251026 14:59:45.073530 22658 raft_consensus.cc:2889] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:45.074898 22729 raft_consensus.cc:1240] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:44.949153 25212 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:45.094141 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:45.061926 25199 raft_consensus.cc:2749] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20251026 14:59:45.101601 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 136350us
I20251026 14:59:45.103960 22729 raft_consensus.cc:1275] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20251026 14:59:45.108557 22808 raft_consensus.cc:2410] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20251026 14:59:45.122471 22726 raft_consensus.cc:2468] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:45.126468 25124 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:45.129043 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.130131 22726 raft_consensus.cc:2393] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 9: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:45.099066 25369 consensus_queue.cc:1048] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20251026 14:59:45.146986 22800 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 9.
W20251026 14:59:45.148566 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 167637us
I20251026 14:59:45.151314 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:45.148413 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:45.064987 25450 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:45.153573 22801 raft_consensus.cc:1275] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251026 14:59:45.154014 22805 raft_consensus.cc:1275] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251026 14:59:45.154635 25348 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:45.154826 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:45.155436 25185 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:45.155900 25196 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:45.156385 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:45.148759 25389 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:45.154147 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:45.157954 22808 raft_consensus.cc:1240] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 8. Current term is 9. Ops: []
I20251026 14:59:45.177697 25389 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.179389 22803 raft_consensus.cc:1275] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:45.179881 25389 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:45.162420 22697 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:45.120977 25436 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.169486 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.169802 25450 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:45.185858 25348 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.169564 25185 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.170885 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.183128 22729 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:45.150044 25405 raft_consensus.cc:2804] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:45.185345 22624 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:45.186322 22655 raft_consensus.cc:1240] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:45.185875 22697 leader_election.cc:400] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Vote denied by peer 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341) with higher term. Message: Invalid argument: T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5.
I20251026 14:59:45.170213 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:45.194785 22697 leader_election.cc:403] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20251026 14:59:45.189255 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:45.186826 25401 raft_consensus.cc:2749] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20251026 14:59:45.229326 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:45.228863 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:45.233026 25124 raft_consensus.cc:2749] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:45.229012 25212 raft_consensus.cc:3060] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:45.234714 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:45.236374 22726 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:45.186335 25436 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:45.256232 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 71012.8us
I20251026 14:59:45.237886 25405 raft_consensus.cc:697] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:45.262581 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:45.238260 25287 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:45.283974 22808 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 10.
I20251026 14:59:45.237604 25196 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.287138 22697 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:45.250038 25429 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 5 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:45.257115 25193 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:45.290077 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:45.292052 22624 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:45.263363 25389 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:45.305486 25124 raft_consensus.cc:2804] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:45.303318 25390 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:45.306073 25124 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:45.306563 25124 raft_consensus.cc:3060] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:45.306821 25405 consensus_queue.cc:237] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.245632 25398 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:45.316440 22697 leader_election.cc:304] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
W20251026 14:59:45.317765 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:45.319454 25398 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Requested vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
W20251026 14:59:45.320218 25493 log.cc:927] Time spent T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
W20251026 14:59:45.321379 25391 consensus_meta.cc:287] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.604s user 0.012s sys 0.000s
I20251026 14:59:45.314782 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:45.321712 22732 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:45.321960 25041 consensus_queue.cc:1059] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 14:59:45.314827 25241 raft_consensus.cc:2749] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:45.333256 22733 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:45.314955 25348 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:45.291769 25287 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.299909 25193 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.302775 25389 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.340272 22622 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:45.336062 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.342772 22697 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:45.343874 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 66964.8us
I20251026 14:59:45.345506 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.334403 25444 raft_consensus.cc:2749] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:45.341549 25041 raft_consensus.cc:3055] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 LEADER]: Stepping down as leader of term 10
I20251026 14:59:45.330520 25391 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.363649 22697 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:45.365345 25401 raft_consensus.cc:2749] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:45.358781 25185 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:45.382959 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.359283 25041 raft_consensus.cc:740] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:59:45.385383 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:45.386498 22727 raft_consensus.cc:2468] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:45.361466 25241 raft_consensus.cc:2749] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251026 14:59:45.386142 22734 raft_consensus.cc:2410] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20251026 14:59:45.374961 25369 consensus_queue.cc:1059] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20251026 14:59:45.395641 22732 raft_consensus.cc:1275] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 11 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 11 index: 5. (index mismatch)
I20251026 14:59:45.401785 22622 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:45.404896 22733 raft_consensus.cc:2468] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:45.414000 22807 raft_consensus.cc:2468] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:45.421329 22697 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:45.419039 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.423787 22697 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:45.425577 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.426499 22732 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:45.419978 25185 raft_consensus.cc:3055] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:45.439807 22774 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:45.423741 25196 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:45.442469 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:45.442497 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:45.442808 25436 raft_consensus.cc:2749] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:45.428893 25241 raft_consensus.cc:2804] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:45.439882 22734 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:45.446596 22732 raft_consensus.cc:3060] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:45.434239 25444 raft_consensus.cc:2804] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:45.437331 25255 raft_consensus.cc:2749] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:45.436080 25185 raft_consensus.cc:740] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:45.438917 25193 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:45.452195 25369 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:45.452785 25241 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:45.445314 25287 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:45.471530 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 75622.6us
W20251026 14:59:45.458990 25498 log.cc:927] Time spent T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
I20251026 14:59:45.453526 25196 raft_consensus.cc:2749] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251026 14:59:45.473639 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.472040 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.477260 22734 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:45.479529 22727 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 9.
I20251026 14:59:45.482962 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.483764 22733 raft_consensus.cc:2393] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 4: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:45.467329 25444 raft_consensus.cc:697] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:45.437583 25041 consensus_queue.cc:260] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.472429 25389 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:45.474494 25241 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:45.478399 25391 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:45.488545 25458 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.038s sys 0.086s
W20251026 14:59:45.482550 25497 log.cc:927] Time spent T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20251026 14:59:45.497012 22732 raft_consensus.cc:2468] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 6.
I20251026 14:59:45.509236 25185 consensus_queue.cc:260] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.516607 25185 raft_consensus.cc:3060] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:45.517813 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:45.519109 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.473485 25369 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.525485 22733 raft_consensus.cc:2468] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:45.525902 23369 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:45.526051 22622 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:45.508437 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:45.508112 25458 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.549s user 0.038s sys 0.087s
I20251026 14:59:45.526763 22655 raft_consensus.cc:1275] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20251026 14:59:45.526826 22774 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:45.518918 25444 consensus_queue.cc:237] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.528532 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:45.528870 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:45.531816 22727 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:45.532333 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:45.533766 25095 raft_consensus.cc:2804] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:45.534955 25255 consensus_queue.cc:1059] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20251026 14:59:45.534687 25041 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Advancing to term 11
I20251026 14:59:45.531177 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:45.535492 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.556113 22774 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:45.556928 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:45.561941 22659 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:45.562865 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:45.557631 22773 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:45.569901 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:45.571437 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:45.530754 25369 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:45.572674 22653 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 6.
I20251026 14:59:45.576122 22622 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:45.574371 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:45.558964 25391 raft_consensus.cc:2749] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251026 14:59:45.528163 25212 raft_consensus.cc:2749] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341) with higher term. Message: Invalid argument: T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5.
I20251026 14:59:45.533483 25436 raft_consensus.cc:2804] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:45.583032 22655 raft_consensus.cc:1275] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20251026 14:59:45.507031 25506 consensus_queue.cc:1059] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.072s
I20251026 14:59:45.577595 22773 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:45.555743 25095 raft_consensus.cc:697] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 LEADER]: Becoming Leader. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:45.582022 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:45.583734 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:45.547644 25429 consensus_queue.cc:1048] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20251026 14:59:45.590673 22654 raft_consensus.cc:2393] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 7: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:45.536484 25405 consensus_queue.cc:1048] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20251026 14:59:45.533067 25124 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.581753 23370 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:45.579267 25196 raft_consensus.cc:2749] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251026 14:59:45.558377 25389 consensus_queue.cc:1059] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20251026 14:59:45.537469 25395 consensus_queue.cc:1048] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20251026 14:59:45.580755 24740 raft_consensus.cc:2804] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:45.601791 22654 raft_consensus.cc:1240] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:45.596460 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:45.598410 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:45.597781 22650 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
I20251026 14:59:45.577631 25510 consensus_queue.cc:1048] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.030s
I20251026 14:59:45.617368 25515 raft_consensus.cc:2804] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:45.623533 25458 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:45.608587 25495 log.cc:927] Time spent T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.167s user 0.001s sys 0.000s
I20251026 14:59:45.630155 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:45.632908 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:45.633572 22774 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:45.601285 24740 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:45.614173 25369 raft_consensus.cc:2804] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:45.640025 22773 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:45.641698 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 65666.3us
I20251026 14:59:45.645367 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.646127 22773 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:45.646732 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:45.647164 22732 raft_consensus.cc:1240] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 11 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20251026 14:59:45.648458 22622 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:45.651971 25507 log.cc:927] Time spent T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20251026 14:59:45.652933 22730 raft_consensus.cc:1240] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:45.653478 25214 raft_consensus.cc:2804] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251026 14:59:45.653893 25214 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:45.653991 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:45.654260 25214 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:45.657625 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.658459 25124 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:45.658969 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:45.660156 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:45.660362 25458 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.661494 25505 raft_consensus.cc:3055] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:45.661929 25505 raft_consensus.cc:740] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:45.662693 25458 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:45.665099 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:45.665766 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 88906.4us
I20251026 14:59:45.666684 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:45.667094 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:45.667482 22657 raft_consensus.cc:2393] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 5: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:45.667980 22653 raft_consensus.cc:2393] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 8: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:45.668489 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:45.668857 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:45.669756 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:45.669899 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:45.641920 25369 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:45.669171 22657 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:45.672317 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:45.659109 22650 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:45.650048 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:45.681232 22725 raft_consensus.cc:2468] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 5.
I20251026 14:59:45.681840 22728 raft_consensus.cc:2410] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20251026 14:59:45.682686 22773 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:45.674086 22700 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:45.661068 25095 consensus_queue.cc:237] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:45.666011 22654 raft_consensus.cc:2393] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 9: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
W20251026 14:59:45.675383 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:45.675693 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:45.676786 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 76995.5us
I20251026 14:59:45.695878 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.679212 22659 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 9.
I20251026 14:59:45.684700 22651 raft_consensus.cc:2393] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 8: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:45.684532 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:45.642505 25348 raft_consensus.cc:2804] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:45.639110 24740 raft_consensus.cc:3060] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:45.629827 25515 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:45.651433 25255 raft_consensus.cc:2749] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:45.703152 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:45.706271 22653 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:45.709894 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:45.710764 22733 raft_consensus.cc:2468] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:45.713021 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:45.715508 22700 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:45.685178 25369 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:45.720461 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:45.697682 25458 raft_consensus.cc:2804] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:45.697623 25215 raft_consensus.cc:2804] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20251026 14:59:45.674156 25390 consensus_queue.cc:1048] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.035s
I20251026 14:59:45.726470 22700 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:45.701819 25348 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:45.725629 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:45.734977 22624 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:45.729533 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:45.639647 25513 consensus_queue.cc:1048] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.084s
I20251026 14:59:45.736320 22733 raft_consensus.cc:2393] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 10: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:45.726995 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:45.732620 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:45.738448 25504 log.cc:927] Time spent T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.251s user 0.002s sys 0.002s
I20251026 14:59:45.738960 22725 raft_consensus.cc:1275] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:45.740300 22725 raft_consensus.cc:1275] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251026 14:59:45.746237 22726 raft_consensus.cc:2393] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 5: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:45.748101 22726 raft_consensus.cc:1275] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20251026 14:59:45.730996 23369 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:45.705202 25515 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:45.694916 25241 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.757720 22650 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
I20251026 14:59:45.704527 25510 consensus_queue.cc:1059] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20251026 14:59:45.662894 25505 consensus_queue.cc:260] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.708510 25405 consensus_queue.cc:1048] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.044s
I20251026 14:59:45.730757 25215 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:45.729967 25401 raft_consensus.cc:2804] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader election won for term 9
I20251026 14:59:45.730623 25458 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
W20251026 14:59:45.772433 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 89856.8us
I20251026 14:59:45.754531 22730 raft_consensus.cc:2468] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:45.754230 22726 raft_consensus.cc:1275] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251026 14:59:45.727687 25514 raft_consensus.cc:2804] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Leader election won for term 10
I20251026 14:59:45.743773 25348 raft_consensus.cc:3060] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:45.803721 22730 raft_consensus.cc:1240] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20251026 14:59:45.797430 22624 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:45.746011 25199 raft_consensus.cc:2804] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election won for term 8
W20251026 14:59:45.800595 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 69584us
I20251026 14:59:45.803418 22773 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:45.814900 22726 raft_consensus.cc:1240] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:45.761224 25436 raft_consensus.cc:697] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:45.836328 22726 raft_consensus.cc:1240] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 8. Current term is 9. Ops: []
W20251026 14:59:45.837102 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 182736us
I20251026 14:59:45.836331 22773 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:45.826305 22725 raft_consensus.cc:1240] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 8. Current term is 9. Ops: []
I20251026 14:59:45.843427 22732 raft_consensus.cc:2468] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:45.776546 25516 raft_consensus.cc:2749] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:45.791275 25215 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Advancing to term 11
I20251026 14:59:45.818202 25186 raft_consensus.cc:2804] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:45.849785 22730 raft_consensus.cc:1275] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20251026 14:59:45.852643 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 77888.7us
I20251026 14:59:45.853972 22624 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:45.850874 22774 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:45.799574 25458 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:45.850709 22700 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:45.854373 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 205994us
I20251026 14:59:45.811239 25199 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:45.797195 25255 consensus_queue.cc:1059] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20251026 14:59:45.845219 25518 raft_consensus.cc:2804] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:45.818305 25514 raft_consensus.cc:697] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:45.812439 25185 consensus_queue.cc:1059] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20251026 14:59:45.855859 25186 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:45.861590 25199 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
W20251026 14:59:45.876672 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 64506.6us
I20251026 14:59:45.865806 25124 raft_consensus.cc:3055] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Stepping down as leader of term 8
I20251026 14:59:45.889468 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:45.889742 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:45.891386 22808 raft_consensus.cc:1240] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:45.892983 22807 raft_consensus.cc:1240] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 8. Current term is 9. Ops: []
I20251026 14:59:45.861411 25520 raft_consensus.cc:2749] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:45.873451 25186 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
W20251026 14:59:45.896943 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 56960.9us
I20251026 14:59:45.897418 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:45.870287 25530 raft_consensus.cc:2804] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:45.878069 25390 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:45.904784 22774 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:45.917229 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 50627.1us
I20251026 14:59:45.918944 22773 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:45.881932 25510 raft_consensus.cc:2749] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251026 14:59:45.885810 25124 raft_consensus.cc:740] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:45.922735 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:45.923729 25528 raft_consensus.cc:2804] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:45.924121 25524 raft_consensus.cc:2749] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:45.931852 22727 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 9.
I20251026 14:59:45.929352 25390 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:45.938285 25521 log.cc:927] Time spent T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.078s user 0.001s sys 0.000s
I20251026 14:59:45.933478 22729 raft_consensus.cc:2468] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:45.943550 22731 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:45.947436 22730 raft_consensus.cc:1240] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 8. Current term is 9. Ops: []
I20251026 14:59:45.947826 22622 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:45.952936 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:45.904768 25530 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:45.952731 22654 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe"
dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:46598
I20251026 14:59:45.966336 22654 raft_consensus.cc:606] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:45.955186 22774 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:45.965447 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:45.982353 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:45.956284 25518 raft_consensus.cc:697] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 LEADER]: Becoming Leader. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:45.958684 25528 raft_consensus.cc:697] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 8 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:46.001597 25390 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:45.903954 25436 consensus_queue.cc:237] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:45.960110 25241 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:45.972538 25530 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:46.014850 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:46.015636 22654 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 10.
I20251026 14:59:46.016894 22700 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:46.001109 25523 raft_consensus.cc:2749] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:46.024529 22734 raft_consensus.cc:2468] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:45.980340 25529 consensus_queue.cc:1059] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.105s
W20251026 14:59:46.039942 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 119506us
I20251026 14:59:46.040979 22774 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:46.014511 25524 raft_consensus.cc:2749] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:46.017591 25536 raft_consensus.cc:3055] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:46.012908 25401 raft_consensus.cc:697] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [term 9 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
W20251026 14:59:46.050140 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 131291us
I20251026 14:59:46.023916 25390 raft_consensus.cc:2804] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20251026 14:59:46.056401 25390 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:46.023362 25389 raft_consensus.cc:2804] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election won for term 6
I20251026 14:59:46.056852 22773 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:46.057257 22654 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42"
dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:46598
I20251026 14:59:45.960883 25514 consensus_queue.cc:237] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.057996 22654 raft_consensus.cc:606] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:46.061154 22654 raft_consensus.cc:614] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251026 14:59:46.062028 22773 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:46.064560 22622 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:46.069473 25540 raft_consensus.cc:2749] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:46.010100 25124 consensus_queue.cc:260] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.056870 25390 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Advancing to term 11
I20251026 14:59:46.055632 25536 raft_consensus.cc:740] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:46.072085 25510 raft_consensus.cc:2749] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251026 14:59:46.109165 22657 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 9.
I20251026 14:59:46.066849 25389 raft_consensus.cc:697] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:46.119189 25389 consensus_queue.cc:237] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.062485 25518 consensus_queue.cc:237] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:46.083537 25535 consensus_queue.cc:1048] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.171s
I20251026 14:59:46.106853 25524 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:46.127825 25369 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.128171 25348 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.056733 25401 consensus_queue.cc:237] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.103943 25522 raft_consensus.cc:2804] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:46.130630 25522 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:46.131249 25522 raft_consensus.cc:3060] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:46.131407 24740 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.071157 25124 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:46.114760 22700 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:46.065253 25528 consensus_queue.cc:237] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.102293 25540 raft_consensus.cc:3055] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:46.134346 22807 raft_consensus.cc:1240] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 8. Current term is 9. Ops: []
W20251026 14:59:46.136821 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:46.112114 25533 raft_consensus.cc:2804] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:46.082386 25438 consensus_queue.cc:1059] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20251026 14:59:46.140504 25444 raft_consensus.cc:2804] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader election won for term 9
I20251026 14:59:46.138414 25540 raft_consensus.cc:740] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:46.147475 25533 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:46.123354 25199 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.141625 25510 consensus_queue.cc:1059] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20251026 14:59:46.152974 25348 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:46.168206 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 106275us
I20251026 14:59:46.171797 22773 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:46.172793 25534 raft_consensus.cc:2804] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:46.173205 25534 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:46.173259 22773 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:46.173626 25534 raft_consensus.cc:3060] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:46.174247 25529 raft_consensus.cc:2804] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:46.174610 25529 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:46.174983 25529 raft_consensus.cc:3060] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:46.174937 25543 consensus_queue.cc:1048] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251026 14:59:46.176930 22773 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:46.177778 25543 raft_consensus.cc:2804] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:46.178364 25543 raft_consensus.cc:697] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [term 8 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:46.179502 25543 consensus_queue.cc:237] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.182312 25534 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.184126 22773 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:46.185065 22773 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:46.184473 25534 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:46.186414 22773 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:46.186942 25534 raft_consensus.cc:2804] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:46.187268 22773 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:46.187529 25534 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:46.188264 25534 raft_consensus.cc:3060] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:46.189078 22773 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:46.167024 25533 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:46.188783 25529 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.167745 25444 raft_consensus.cc:697] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 9 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
W20251026 14:59:46.216267 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 175338us
I20251026 14:59:46.221050 22774 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:46.199278 25369 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:46.233796 25548 raft_consensus.cc:2804] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:46.240027 25199 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:46.167233 25536 consensus_queue.cc:260] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.215929 25348 raft_consensus.cc:2804] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20251026 14:59:46.250176 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 59173.2us
I20251026 14:59:46.223476 25535 raft_consensus.cc:2804] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251026 14:59:46.214027 25514 raft_consensus.cc:3055] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Stepping down as leader of term 8
I20251026 14:59:46.198017 25215 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.256713 25529 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:46.229802 24740 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:46.273612 22774 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:46.245594 25510 raft_consensus.cc:3055] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Stepping down as leader of term 7
W20251026 14:59:46.280598 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 195781us
I20251026 14:59:46.245581 25548 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:46.287550 22774 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:46.288801 25545 raft_consensus.cc:2804] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:46.290066 25444 consensus_queue.cc:237] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:46.297410 25547 raft_consensus.cc:2804] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:46.298152 25547 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:46.298625 25215 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 11 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:46.298754 25389 raft_consensus.cc:2749] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:46.253383 25348 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:46.301395 25548 raft_consensus.cc:3060] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:46.299582 25215 raft_consensus.cc:2749] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251026 14:59:46.301906 22773 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:46.302340 25214 consensus_meta.cc:287] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.648s user 0.013s sys 0.000s
I20251026 14:59:46.303040 22773 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:46.303550 25215 raft_consensus.cc:2804] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:46.303969 25215 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:46.304317 25215 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:46.304986 25549 raft_consensus.cc:2804] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader election won for term 9
I20251026 14:59:46.242650 25540 consensus_queue.cc:260] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.302815 25214 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.333698 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:46.334322 25214 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:46.334550 22731 raft_consensus.cc:3060] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:46.338435 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:46.339237 22732 raft_consensus.cc:2393] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 8: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:46.340289 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:46.341002 22732 raft_consensus.cc:3060] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:46.257979 25514 raft_consensus.cc:740] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:59:46.342067 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 99196.8us
I20251026 14:59:46.298580 25547 raft_consensus.cc:3060] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:46.254015 25535 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:46.343220 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 55761us
I20251026 14:59:46.345541 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:46.352746 22726 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:46.286682 25510 raft_consensus.cc:740] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:46.367340 22731 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:46.375448 22773 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:46.272713 25544 raft_consensus.cc:2764] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20251026 14:59:46.357900 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:46.359480 22697 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:46.360256 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:46.380231 22728 raft_consensus.cc:3060] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:46.360723 22733 raft_consensus.cc:1240] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:46.344391 25549 raft_consensus.cc:697] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [term 9 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:46.362308 25530 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.286275 25369 consensus_queue.cc:1048] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.079s
I20251026 14:59:46.301918 25348 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:46.366319 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:46.389911 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:46.390866 22731 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:46.429605 25529 raft_consensus.cc:2804] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:46.430476 25529 raft_consensus.cc:697] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:46.431630 25529 consensus_queue.cc:237] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.432529 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:46.433183 22733 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Advancing to term 11
I20251026 14:59:46.305497 25543 consensus_queue.cc:1059] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
I20251026 14:59:46.345598 25540 raft_consensus.cc:3060] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:46.441958 22728 raft_consensus.cc:2468] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:46.372596 25524 raft_consensus.cc:2749] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:46.451680 22808 raft_consensus.cc:1240] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
W20251026 14:59:46.386047 25458 consensus_meta.cc:287] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.522s user 0.006s sys 0.004s
I20251026 14:59:46.467715 25458 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:46.377573 25535 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:46.469930 25458 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:46.470592 25458 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:46.469810 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:46.437311 25539 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:46.465761 22726 raft_consensus.cc:2468] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 9.
I20251026 14:59:46.471122 25458 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.462908 22773 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:46.477421 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:46.462949 22732 raft_consensus.cc:2468] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:46.481777 22731 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 10.
I20251026 14:59:46.437150 25481 raft_consensus.cc:2749] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251026 14:59:46.503415 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:46.372210 25534 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.536221 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:46.537021 22805 raft_consensus.cc:3060] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:46.538151 25534 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:46.538569 22654 raft_consensus.cc:1275] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20251026 14:59:46.538763 25534 raft_consensus.cc:2804] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:46.538926 22654 pending_rounds.cc:85] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29: Aborting all ops after (but not including) 4
I20251026 14:59:46.539194 22654 pending_rounds.cc:107] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20251026 14:59:46.539273 25534 raft_consensus.cc:697] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:46.539448 22654 raft_consensus.cc:2889] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:46.509168 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:46.540216 25534 consensus_queue.cc:237] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.517534 25549 consensus_queue.cc:237] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.524077 25530 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:46.528764 25522 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:46.545024 22806 raft_consensus.cc:1240] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:46.546109 25522 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:46.371016 25514 consensus_queue.cc:260] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.547272 25514 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
W20251026 14:59:46.533842 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 71053.3us
I20251026 14:59:46.525910 25458 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:46.537532 22697 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:46.553133 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:46.554095 22808 raft_consensus.cc:2468] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
I20251026 14:59:46.554948 22697 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:46.501947 22733 raft_consensus.cc:2468] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 11.
I20251026 14:59:46.552662 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:46.551179 22773 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:46.557241 22773 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:46.556970 22805 raft_consensus.cc:2468] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4.
I20251026 14:59:46.558640 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:46.560834 22697 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:46.370047 25510 consensus_queue.cc:260] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.507555 25539 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.558708 22773 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:46.567934 25539 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:46.563302 25515 consensus_meta.cc:287] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1: Time spent flushing consensus metadata: real 0.794s user 0.013s sys 0.000s
I20251026 14:59:46.569965 25515 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:46.570396 25390 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.572481 25390 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 11 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:46.573660 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:46.574903 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:46.575605 22808 raft_consensus.cc:2393] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 10: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:46.576372 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:46.576889 22808 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:46.580085 22734 raft_consensus.cc:1240] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 11 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
W20251026 14:59:46.553001 25567 log.cc:927] Time spent T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.184s user 0.002s sys 0.002s
I20251026 14:59:46.591292 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:46.592092 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:46.592271 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:46.583078 25398 raft_consensus.cc:2749] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:46.578692 25481 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:46.594022 22697 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:46.578926 25543 raft_consensus.cc:2804] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader election won for term 9
I20251026 14:59:46.579064 25369 raft_consensus.cc:2804] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election won for term 6
W20251026 14:59:46.590484 25186 consensus_meta.cc:287] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.660s user 0.005s sys 0.005s
I20251026 14:59:46.599962 22624 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:46.601169 22657 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d"
dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:46598
I20251026 14:59:46.597723 22697 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:46.583105 25515 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:46.589107 25390 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:46.604609 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:59:46.605621 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:46.594344 23400 raft_consensus.cc:1240] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:46.608950 22805 raft_consensus.cc:2376] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 4. Re-sending same reply.
I20251026 14:59:46.596366 25458 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:46.595829 25540 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:46.614537 22624 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:46.599503 25199 raft_consensus.cc:2804] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:46.611769 22773 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:46.610786 22697 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:46.590771 25215 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.623380 22697 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:46.624917 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:46.625564 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:46.626155 22656 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:46.609397 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:46.645889 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:46.640329 22808 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 9.
I20251026 14:59:46.641322 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:46.613703 25398 raft_consensus.cc:2804] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader election won for term 4
I20251026 14:59:46.646760 22655 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Advancing to term 11
I20251026 14:59:46.612389 25543 raft_consensus.cc:697] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [term 9 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:46.647253 22652 raft_consensus.cc:2468] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:46.602727 25186 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.621601 25199 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:46.652508 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:46.656082 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:46.654647 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:46.658109 22653 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:46.652578 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:46.682163 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:46.642947 25539 raft_consensus.cc:2749] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251026 14:59:46.683411 25570 raft_consensus.cc:2804] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader election won for term 10
I20251026 14:59:46.683605 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:46.684886 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:46.686719 25548 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.689813 25533 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.691720 22728 raft_consensus.cc:1275] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251026 14:59:46.691927 25533 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:46.644424 25353 raft_consensus.cc:2749] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:46.624871 25540 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.652746 25436 raft_consensus.cc:2749] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:46.616140 25458 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.628947 25522 raft_consensus.cc:493] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:46.700750 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:46.703118 22655 raft_consensus.cc:2468] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 11.
I20251026 14:59:46.688956 25548 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:46.696316 25540 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:46.713196 22659 raft_consensus.cc:2468] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
W20251026 14:59:46.717986 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 103515us
I20251026 14:59:46.693993 25546 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:46.772637 25546 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.775369 25546 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:46.778081 22803 raft_consensus.cc:1240] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20251026 14:59:46.706568 25522 raft_consensus.cc:515] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:46.644016 25389 raft_consensus.cc:2804] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 11 FOLLOWER]: Leader election won for term 11
I20251026 14:59:46.694519 25215 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:46.682144 25398 raft_consensus.cc:697] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
W20251026 14:59:46.789978 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:46.795136 22808 raft_consensus.cc:2468] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:46.705226 25481 raft_consensus.cc:2749] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251026 14:59:46.756279 25436 raft_consensus.cc:2749] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251026 14:59:46.782720 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:46.695952 25547 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.813046 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:46.825834 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:46.697305 25532 consensus_queue.cc:1048] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.116s
I20251026 14:59:46.776322 25570 raft_consensus.cc:697] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 10 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:46.681005 25199 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:46.829770 22659 raft_consensus.cc:1275] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
W20251026 14:59:46.833779 25564 log.cc:927] Time spent T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.464s user 0.002s sys 0.000s
I20251026 14:59:46.833999 25547 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:46.835949 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:46.837383 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:46.837901 25560 log.cc:927] Time spent T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.468s user 0.000s sys 0.002s
I20251026 14:59:46.838001 22800 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
I20251026 14:59:46.839048 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:46.813849 25539 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:46.840848 25539 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:46.841262 25565 log.cc:927] Time spent T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1: Append to log took a long time: real 0.468s user 0.002s sys 0.000s
I20251026 14:59:46.842810 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1201) took 1202 ms. Trace:
I20251026 14:59:46.842932 25539 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:46.843231 22725 rpcz_store.cc:276] 1026 14:59:45.640895 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:45.740872 (+ 99977us) service_pool.cc:225] Handling call
1026 14:59:45.741130 (+ 258us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:45.839556 (+ 98426us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:46.842574 (+1003018us) spinlock_profiling.cc:243] Waited 68.9 ms on lock 0x7b54002db754. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:46.842749 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":158485376}
I20251026 14:59:46.845033 22801 raft_consensus.cc:2393] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 7: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:46.827972 25546 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:46.846760 22658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32888 (request call id 142) took 1319 ms. Trace:
I20251026 14:59:46.846696 25546 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:46.847738 25558 log.cc:927] Time spent T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.484s user 0.001s sys 0.000s
I20251026 14:59:46.695361 25543 consensus_queue.cc:237] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.849489 25436 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:46.850004 25405 consensus_queue.cc:1048] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 14:59:46.837400 25458 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:46.846096 25186 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:46.852825 23367 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:46.835785 25517 raft_consensus.cc:3055] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:46.871196 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:46.872118 22808 raft_consensus.cc:2468] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:46.847236 22658 rpcz_store.cc:276] 1026 14:59:45.527791 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:45.528625 (+ 834us) service_pool.cc:225] Handling call
1026 14:59:46.846010 (+1317385us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b5800000998. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8ce074451 00007fd8ce074302 00007fd8d6f0dad9 00007fd8d6090d30 00007fd8d615984b 00007fd8d61598d1 00007fd8d611b580 00007fd8d6105ee9 00007fd8d6105992 00007fd8cf69fd34
1026 14:59:46.846220 (+ 210us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:46.846449 (+ 229us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:46.846475 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:46.846516 (+ 41us) raft_consensus.cc:1645] Marking committed up to 6
1026 14:59:46.846556 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:46.846612 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:46.846706 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24835840}
I20251026 14:59:46.862073 22697 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:46.851464 25398 consensus_queue.cc:237] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.869997 22653 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 9.
I20251026 14:59:46.849005 25546 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:46.822161 25389 raft_consensus.cc:697] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 11 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:46.878898 22700 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:46.878029 22800 raft_consensus.cc:1275] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20251026 14:59:46.887866 22807 raft_consensus.cc:1275] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20251026 14:59:46.888324 22807 pending_rounds.cc:85] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821: Aborting all ops after (but not including) 4
I20251026 14:59:46.888645 22807 pending_rounds.cc:107] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20251026 14:59:46.888927 22807 raft_consensus.cc:2889] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:46.884604 22697 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:46.693245 25348 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.822412 25369 raft_consensus.cc:697] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:46.888995 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:46.892685 22802 raft_consensus.cc:3060] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:46.873948 25532 raft_consensus.cc:2749] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251026 14:59:46.894888 22730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1216) took 1018 ms. Trace:
I20251026 14:59:46.831012 25444 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:46.871786 25517 raft_consensus.cc:740] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:46.900149 22656 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
I20251026 14:59:46.900760 22808 raft_consensus.cc:1275] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251026 14:59:46.911509 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:46.901401 25353 raft_consensus.cc:2749] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251026 14:59:46.902040 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:46.870213 25390 raft_consensus.cc:2749] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:46.903817 22730 rpcz_store.cc:276] 1026 14:59:45.876394 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:45.876739 (+ 345us) service_pool.cc:225] Handling call
1026 14:59:45.947249 (+ 70510us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:45.961762 (+ 14513us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:46.894670 (+932908us) spinlock_profiling.cc:243] Waited 70.1 ms on lock 0x7b540008cfd0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:46.894822 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":161228544}
I20251026 14:59:46.879395 25530 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:46.897197 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 128353us
I20251026 14:59:46.913595 22804 raft_consensus.cc:1240] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:46.916502 25522 leader_election.cc:290] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:46.850015 25436 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.925698 22624 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:46.927907 22807 raft_consensus.cc:1275] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20251026 14:59:46.846031 25570 consensus_queue.cc:237] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.932770 22800 raft_consensus.cc:1275] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20251026 14:59:46.933926 22800 raft_consensus.cc:3055] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:46.934262 22800 raft_consensus.cc:740] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:46.935416 22800 consensus_queue.cc:260] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.936579 22800 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:46.948216 22624 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:46.920073 25539 raft_consensus.cc:2804] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader election won for term 5
W20251026 14:59:46.954972 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 70454.7us
I20251026 14:59:46.917834 25583 raft_consensus.cc:2804] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:46.956419 22697 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:46.957620 22697 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:46.961354 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 82541.9us
I20251026 14:59:46.962445 22700 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:46.912544 25444 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.952648 25514 raft_consensus.cc:2749] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20251026 14:59:46.974138 25587 raft_consensus.cc:2749] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251026 14:59:46.958657 25389 consensus_queue.cc:237] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.926436 25535 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.952296 25580 consensus_queue.cc:1048] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.029s
I20251026 14:59:46.959465 25583 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:46.930173 25530 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:46.947196 25586 consensus_queue.cc:1059] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.082s
I20251026 14:59:46.953945 25517 consensus_queue.cc:260] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:47.038774 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:47.047899 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 85539.8us
I20251026 14:59:47.048813 22700 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:47.023085 22802 raft_consensus.cc:1275] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20251026 14:59:47.051661 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 94085.4us
I20251026 14:59:47.052335 22697 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:46.958639 25390 raft_consensus.cc:2749] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20251026 14:59:47.053817 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 105653us
I20251026 14:59:47.054937 25517 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:47.077452 22653 raft_consensus.cc:1275] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 5. (term mismatch)
I20251026 14:59:47.081804 22624 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:47.082001 22653 pending_rounds.cc:85] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29: Aborting all ops after (but not including) 4
I20251026 14:59:47.082473 22653 pending_rounds.cc:107] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29: Aborting uncommitted NO_OP operation due to leader change: 10.5
I20251026 14:59:47.060776 22700 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:47.082821 22653 raft_consensus.cc:2889] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:46.979138 25458 raft_consensus.cc:2804] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 11 FOLLOWER]: Leader election won for term 11
I20251026 14:59:47.085649 22700 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:46.979339 25583 raft_consensus.cc:3060] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:47.091971 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:47.088294 25530 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:46.979744 25481 raft_consensus.cc:2804] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251026 14:59:47.088753 25390 consensus_queue.cc:1048] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:59:46.964092 25539 raft_consensus.cc:697] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:47.104636 22728 raft_consensus.cc:1275] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251026 14:59:47.104866 25539 consensus_queue.cc:237] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:47.105792 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:47.106468 22728 raft_consensus.cc:2468] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:47.107717 22728 raft_consensus.cc:1275] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:47.053488 25348 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:47.109524 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:47.110342 22727 raft_consensus.cc:2393] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 8: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
W20251026 14:59:47.093473 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:47.117482 22730 raft_consensus.cc:1240] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:47.022713 25570 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:47.093765 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:47.095439 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:47.123919 22733 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:47.096014 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:47.125825 22731 raft_consensus.cc:2393] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 7: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:47.097676 22732 raft_consensus.cc:1240] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 8. Current term is 9. Ops: []
I20251026 14:59:47.054507 25353 raft_consensus.cc:2749] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251026 14:59:47.031594 25444 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:47.030570 25436 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:47.099550 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:47.192510 22653 raft_consensus.cc:2468] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:47.088163 25591 raft_consensus.cc:2804] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader election won for term 9
I20251026 14:59:46.980312 25573 consensus_queue.cc:1059] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.127s
I20251026 14:59:47.099202 25481 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:47.203555 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:47.204298 22727 raft_consensus.cc:2393] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 9: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:47.103332 25307 consensus_queue.cc:1059] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20251026 14:59:46.983814 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:47.091102 25535 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:47.192674 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:47.050762 25505 raft_consensus.cc:3060] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:47.034977 25199 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:47.204088 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:47.103525 25592 raft_consensus.cc:2804] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:47.205864 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:47.208735 22732 raft_consensus.cc:1240] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:47.208616 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:47.202586 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:47.024163 25588 consensus_queue.cc:1048] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.163s
I20251026 14:59:46.984014 25369 consensus_queue.cc:237] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:47.219393 22731 raft_consensus.cc:1275] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251026 14:59:47.120317 25458 raft_consensus.cc:697] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 11 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
W20251026 14:59:47.228152 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 160194us
I20251026 14:59:47.226016 22730 raft_consensus.cc:3060] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:47.191567 25390 raft_consensus.cc:2804] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:47.227226 22773 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:47.230050 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:47.230574 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 148831us
I20251026 14:59:47.225946 22728 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:47.229050 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:47.378048 22727 raft_consensus.cc:2468] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:47.231401 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:47.380581 22653 raft_consensus.cc:3060] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:47.383760 25556 consensus_queue.cc:1048] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.318s
W20251026 14:59:47.387140 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 160027us
I20251026 14:59:47.209367 25481 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:47.391233 22728 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:47.391379 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:47.392885 22732 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:47.363454 22654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1363) took 1254 ms. Trace:
I20251026 14:59:47.398460 22733 raft_consensus.cc:2468] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:47.397158 22654 rpcz_store.cc:276] 1026 14:59:46.108947 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:46.109179 (+ 232us) service_pool.cc:225] Handling call
1026 14:59:46.118677 (+ 9498us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:46.539904 (+421227us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:47.232615 (+692711us) spinlock_profiling.cc:243] Waited 419 ms on lock 0x7b540009ddd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:47.363357 (+130742us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":964786944}
I20251026 14:59:47.230444 25592 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:47.226516 25573 consensus_queue.cc:1243] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer 0d684445287b450a8f738b6ad4b59821 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20251026 14:59:47.230234 25540 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:47.222672 25353 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:47.231752 25591 raft_consensus.cc:697] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 9 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:47.196071 25570 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:47.228646 25575 raft_consensus.cc:2804] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:47.403025 25212 consensus_queue.cc:1048] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20251026 14:59:47.431133 22730 raft_consensus.cc:2468] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:47.437722 22732 raft_consensus.cc:2468] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 10.
I20251026 14:59:47.438094 22653 raft_consensus.cc:2468] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:47.439219 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:47.228899 25534 consensus_queue.cc:1048] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.189s
I20251026 14:59:47.450733 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:47.452009 22654 raft_consensus.cc:3060] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:47.403352 25546 raft_consensus.cc:2804] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:47.410586 25592 raft_consensus.cc:3060] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:47.471894 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:47.482954 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:47.447317 25573 raft_consensus.cc:3055] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Stepping down as leader of term 8
I20251026 14:59:47.570685 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:47.431793 25390 raft_consensus.cc:697] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 8 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:47.403764 25535 consensus_queue.cc:1048] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20251026 14:59:47.457388 25546 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:47.422187 25540 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:47.428257 25353 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:47.663843 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:47.668871 22653 raft_consensus.cc:2393] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 6: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:47.669760 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:47.491117 25522 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:47.670496 22653 raft_consensus.cc:2393] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 8: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:47.461189 25348 consensus_queue.cc:1059] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.206s
I20251026 14:59:47.570221 25573 raft_consensus.cc:740] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:47.687428 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:47.664590 25546 raft_consensus.cc:3060] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:47.570140 25530 consensus_queue.cc:1059] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.109s
I20251026 14:59:47.665297 25444 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:47.697369 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:47.671047 25575 raft_consensus.cc:697] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:47.707202 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
W20251026 14:59:47.705865 25196 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
W20251026 14:59:47.711176 25583 consensus_meta.cc:287] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.607s user 0.000s sys 0.009s
I20251026 14:59:47.658946 25458 consensus_queue.cc:237] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:47.678676 25522 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:47.720590 25199 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:47.743191 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:47.709263 25444 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:47.746819 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 371334us
I20251026 14:59:47.757237 25522 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:47.762220 25545 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:47.762982 25545 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:47.768193 25544 raft_consensus.cc:3055] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:47.768870 25544 raft_consensus.cc:740] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:47.770022 25544 consensus_queue.cc:260] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:47.771152 25572 log.cc:927] Time spent T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20251026 14:59:47.771525 25544 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
W20251026 14:59:47.770057 25603 log.cc:927] Time spent T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20251026 14:59:47.775290 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:47.779234 25444 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:47.780787 22652 raft_consensus.cc:1240] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:47.734048 25583 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:47.782064 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:47.782811 22652 raft_consensus.cc:2393] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 8: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:47.783507 25583 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:47.783716 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:47.788271 22804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1450) took 1474 ms. Trace:
I20251026 14:59:47.788671 22804 rpcz_store.cc:276] 1026 14:59:46.313655 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:46.313809 (+ 154us) service_pool.cc:225] Handling call
1026 14:59:46.343556 (+ 29747us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:46.924324 (+580768us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:47.787985 (+863661us) spinlock_profiling.cc:243] Waited 570 ms on lock 0x7b540004bfd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:47.788185 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1310326656}
I20251026 14:59:47.784327 22652 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:47.791107 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:47.799798 22624 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:47.736743 25398 raft_consensus.cc:493] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:47.789151 25570 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:47.702633 25591 consensus_queue.cc:237] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:47.805310 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:47.809638 25353 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:47.744602 25607 consensus_queue.cc:1048] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
I20251026 14:59:47.813266 22653 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 9.
I20251026 14:59:47.814625 25611 consensus_queue.cc:1059] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:59:47.730253 25390 consensus_queue.cc:237] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:47.820281 22554 catalog_manager.cc:5649] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "248493ec312e4398b828c6cb00e5c4b1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 14:59:47.821527 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:47.821983 22554 catalog_manager.cc:5649] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 reported cstate change: config changed from index 2 to 6, 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) changed from NON_VOTER to VOTER. New cstate: current_term: 6 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } } }
I20251026 14:59:47.822881 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:47.824533 25353 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:47.825060 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40908 (request call id 150) took 1624 ms. Trace:
I20251026 14:59:47.825032 25353 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:47.810746 25398 raft_consensus.cc:515] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:47.825701 23400 rpcz_store.cc:276] 1026 14:59:46.200766 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:46.204027 (+ 3261us) service_pool.cc:225] Handling call
1026 14:59:46.225790 (+ 21763us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:46.613905 (+388115us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:47.824738 (+1210833us) spinlock_profiling.cc:243] Waited 354 ms on lock 0x7b540055a5d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:47.824957 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":814124288}
I20251026 14:59:47.827582 25398 leader_election.cc:290] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:47.820669 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1600) took 1516 ms. Trace:
I20251026 14:59:47.824095 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:47.827607 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:47.831732 22656 raft_consensus.cc:2368] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 8. Current term is 9.
I20251026 14:59:47.833586 22654 raft_consensus.cc:2468] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:47.834897 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:47.830003 22734 rpcz_store.cc:276] 1026 14:59:46.304363 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:46.304537 (+ 174us) service_pool.cc:225] Handling call
1026 14:59:46.304903 (+ 366us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:46.582698 (+277795us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:47.820377 (+1237679us) spinlock_profiling.cc:243] Waited 275 ms on lock 0x7b54002dc154. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:47.820601 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":632329600}
I20251026 14:59:47.839349 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:47.841077 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:47.802132 25610 raft_consensus.cc:3055] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 8 LEADER]: Stepping down as leader of term 8
W20251026 14:59:47.811069 25596 log.cc:927] Time spent T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.417s user 0.001s sys 0.000s
I20251026 14:59:47.827644 25353 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:47.852368 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:47.727437 25573 consensus_queue.cc:260] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:47.860812 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 61122.2us
W20251026 14:59:47.861011 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 467713us
I20251026 14:59:47.862560 22624 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
W20251026 14:59:47.854660 25554 log.cc:927] Time spent T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1: Append to log took a long time: real 1.491s user 0.013s sys 0.005s
I20251026 14:59:47.858793 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:47.882612 22808 raft_consensus.cc:1240] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 5. Current term is 6. Ops: []
W20251026 14:59:47.877074 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 307425us
I20251026 14:59:47.885903 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:47.859860 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:47.813032 25540 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:47.873137 22697 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:47.887650 22656 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:47.803539 25481 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:47.888509 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:47.907426 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:47.904616 22651 raft_consensus.cc:2468] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:47.908242 22659 raft_consensus.cc:2468] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:47.900795 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:47.844820 25610 raft_consensus.cc:740] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:47.914983 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:47.918133 22802 raft_consensus.cc:1240] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:47.918850 22800 raft_consensus.cc:1240] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20251026 14:59:47.919519 22807 raft_consensus.cc:1240] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:47.911439 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:47.928407 22807 raft_consensus.cc:1240] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:47.838555 25575 consensus_queue.cc:237] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:47.929459 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:47.930099 22800 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:47.932139 22802 raft_consensus.cc:1275] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251026 14:59:47.932413 22800 raft_consensus.cc:1240] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
W20251026 14:59:47.938161 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 59124.7us
I20251026 14:59:47.939620 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:47.942881 22804 raft_consensus.cc:1240] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [term 11 LEADER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 10. Current term is 11. Ops: []
I20251026 14:59:47.873744 25607 consensus_queue.cc:1059] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.053s
W20251026 14:59:47.926030 25596 log.cc:927] Time spent T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20251026 14:59:47.953181 22651 raft_consensus.cc:1240] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 4. Current term is 5. Ops: []
W20251026 14:59:47.953858 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 91363us
I20251026 14:59:47.940413 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:47.958750 22654 raft_consensus.cc:1240] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 8. Current term is 9. Ops: []
I20251026 14:59:47.955113 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:47.977486 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:47.950534 25241 consensus_queue.cc:1048] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.083s
I20251026 14:59:48.002471 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:48.027616 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:48.028280 22804 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
W20251026 14:59:48.010480 23367 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.002s sys 0.010s
W20251026 14:59:48.029987 23367 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.596s user 0.003s sys 0.010s
I20251026 14:59:48.017706 22800 raft_consensus.cc:2468] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:48.017144 22624 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:48.032997 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:48.021847 22806 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.001s sys 0.012s
W20251026 14:59:48.001619 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:48.023072 22774 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:48.002815 22659 raft_consensus.cc:1275] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20251026 14:59:47.972079 22655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1589) took 1009 ms. Trace:
I20251026 14:59:48.036537 25199 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:48.021669 25539 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:48.037919 25604 raft_consensus.cc:2804] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election won for term 8
W20251026 14:59:48.034034 22806 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.608s user 0.001s sys 0.012s
I20251026 14:59:48.016709 22807 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:48.024410 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1593) took 1047 ms. Trace:
I20251026 14:59:48.046481 22655 rpcz_store.cc:276] 1026 14:59:46.962772 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:46.962901 (+ 129us) service_pool.cc:225] Handling call
1026 14:59:47.058451 (+ 95550us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:47.409585 (+351134us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:47.409636 (+ 51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:47.422635 (+ 12999us) log.cc:844] Serialized 26 byte log entry
1026 14:59:47.681794 (+259159us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:47.911071 (+229277us) log.cc:844] Serialized 14 byte log entry
1026 14:59:47.971639 (+ 60568us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:47.971747 (+ 108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:47.971808 (+ 61us) raft_consensus.cc:1685] finished
1026 14:59:47.971827 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:47.971999 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":60080,"raft.queue_time_us":240171,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":25384,"spinlock_wait_cycles":60544,"thread_start_us":725672,"threads_started":3,"wal-append.queue_time_us":350112}
I20251026 14:59:48.065320 22774 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:48.017383 22653 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 9.
I20251026 14:59:47.938030 25583 consensus_queue.cc:1048] T 988275d9b2574d0381d8f21ee8d99381 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.097s
W20251026 14:59:48.031302 22604 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20251026 14:59:47.953878 25255 raft_consensus.cc:2804] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:47.929754 25390 raft_consensus.cc:2749] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:48.049456 22659 raft_consensus.cc:1275] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251026 14:59:48.059640 22651 rpcz_store.cc:276] 1026 14:59:46.976712 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:47.920714 (+944002us) service_pool.cc:225] Handling call
1026 14:59:47.952947 (+ 32233us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:48.024076 (+ 71129us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:48.024336 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:48.051627 25505 consensus_meta.cc:287] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.823s user 0.011s sys 0.000s
I20251026 14:59:48.070540 22774 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:48.094830 22700 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.878s user 0.013s sys 0.027s
W20251026 14:59:48.099143 22700 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.882s user 0.014s sys 0.028s
I20251026 14:59:48.079418 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:48.108131 22804 raft_consensus.cc:2393] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 6: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:47.938232 25444 consensus_queue.cc:1048] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.122s
W20251026 14:59:48.152371 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 81940.4us
W20251026 14:59:48.147114 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:48.147533 25627 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:48.155072 25519 raft_consensus.cc:2804] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:48.155295 25255 raft_consensus.cc:697] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:59:48.155506 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 105929us
W20251026 14:59:48.156510 25528 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.794s user 0.004s sys 0.014s
I20251026 14:59:48.058739 23400 raft_consensus.cc:1240] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 LEADER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:48.157681 22651 raft_consensus.cc:1240] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20251026 14:59:48.161417 22659 raft_consensus.cc:1275] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:48.162618 22774 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:48.163161 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:48.178359 22800 raft_consensus.cc:1240] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:48.156430 25604 raft_consensus.cc:697] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:47.950829 25598 consensus_queue.cc:1048] T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.147s
I20251026 14:59:48.179800 22807 raft_consensus.cc:2468] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 9.
I20251026 14:59:47.957960 25613 consensus_queue.cc:1048] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.210s
I20251026 14:59:48.041764 25193 consensus_queue.cc:1048] T 627ebd70757a47b18827c38f071b5261 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.116s
I20251026 14:59:48.098273 25398 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:48.068217 25539 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:48.185417 22654 raft_consensus.cc:1240] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:48.121210 25348 raft_consensus.cc:2749] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20251026 14:59:48.182229 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:48.177919 25528 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.816s user 0.005s sys 0.015s
I20251026 14:59:48.058092 25199 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.216367 22654 raft_consensus.cc:1275] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20251026 14:59:48.174789 25196 raft_consensus.cc:2749] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20251026 14:59:48.204617 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:48.199198 22774 leader_election.cc:400] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 8. Current term is 9.
I20251026 14:59:48.227303 22774 leader_election.cc:403] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20251026 14:59:48.218964 22654 raft_consensus.cc:1275] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:48.214114 25604 consensus_queue.cc:237] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.222311 25618 raft_consensus.cc:2749] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251026 14:59:48.222792 25609 log.cc:927] Time spent T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
I20251026 14:59:48.223133 25599 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:48.223534 25559 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:48.183825 25520 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:48.223631 25545 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:48.252599 25628 log.cc:927] Time spent T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.219s user 0.000s sys 0.001s
I20251026 14:59:48.244215 22734 raft_consensus.cc:1275] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20251026 14:59:48.215978 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:48.242072 22651 raft_consensus.cc:1275] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251026 14:59:48.251395 25555 consensus_queue.cc:1059] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20251026 14:59:48.248068 22654 raft_consensus.cc:3055] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 LEADER]: Stepping down as leader of term 9
I20251026 14:59:48.267489 22652 raft_consensus.cc:2468] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
W20251026 14:59:48.273514 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 76196.7us
I20251026 14:59:48.181823 25570 consensus_queue.cc:1059] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.057s
I20251026 14:59:48.234318 25348 raft_consensus.cc:2749] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251026 14:59:48.175912 25627 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:48.280920 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 98198.5us
I20251026 14:59:48.295809 22773 leader_election.cc:304] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:48.296079 25627 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:48.297101 22773 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:48.279379 22654 raft_consensus.cc:740] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:48.282648 22774 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:48.286581 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 142326us
I20251026 14:59:48.272442 25559 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.293005 25255 consensus_queue.cc:237] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.306959 22774 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:48.307184 22773 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:48.309079 22773 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:48.283890 25546 consensus_meta.cc:287] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.572s user 0.008s sys 0.003s
I20251026 14:59:48.312701 22732 raft_consensus.cc:1240] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 9 LEADER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 8. Current term is 9. Ops: []
I20251026 14:59:48.224678 25398 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.278716 25481 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:48.319865 25348 raft_consensus.cc:2749] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 14:59:48.328063 25348 raft_consensus.cc:2804] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:48.328491 25348 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:48.328912 25348 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:48.331957 25199 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:48.320106 25546 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.272070 25196 consensus_queue.cc:1048] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.036s
W20251026 14:59:48.318903 25544 consensus_meta.cc:287] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.547s user 0.010s sys 0.002s
I20251026 14:59:48.350281 22803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1445) took 2361 ms. Trace:
I20251026 14:59:48.058914 25610 consensus_queue.cc:260] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.319029 25519 raft_consensus.cc:697] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:48.318311 25615 raft_consensus.cc:2749] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185) with higher term. Message: Invalid argument: T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 8. Current term is 9.
I20251026 14:59:48.360811 25618 raft_consensus.cc:2804] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:48.313658 25516 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
W20251026 14:59:48.361202 25551 log.cc:927] Time spent T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 2.000s user 0.002s sys 0.048s
W20251026 14:59:48.361370 25630 log.cc:927] Time spent T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.096s user 0.002s sys 0.000s
I20251026 14:59:48.288398 25613 consensus_queue.cc:1059] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20251026 14:59:48.361785 25193 raft_consensus.cc:2804] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:48.362010 25636 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:48.326687 25570 raft_consensus.cc:2749] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:48.355814 22803 rpcz_store.cc:276] 1026 14:59:45.962354 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:45.962490 (+ 136us) service_pool.cc:225] Handling call
1026 14:59:45.962905 (+ 415us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:46.784448 (+821543us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:48.323684 (+1539236us) spinlock_profiling.cc:243] Waited 815 ms on lock 0x7b54003345d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:48.323890 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1873926016}
W20251026 14:59:48.359025 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 49999.9us
I20251026 14:59:48.413960 22773 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:48.350368 22654 consensus_queue.cc:260] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.318748 25627 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:48.375550 25401 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:48.377477 25398 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:48.377797 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:48.365792 25199 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:48.266480 25599 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.379753 22730 raft_consensus.cc:1275] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20251026 14:59:48.334395 25591 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:48.381644 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:48.382288 22802 raft_consensus.cc:2368] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 6. Current term is 7.
I20251026 14:59:48.383354 22700 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:48.269009 25520 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:48.370317 25592 consensus_meta.cc:287] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.751s user 0.005s sys 0.006s
W20251026 14:59:48.382453 25517 consensus_meta.cc:287] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 1.288s user 0.008s sys 0.005s
I20251026 14:59:48.379161 22726 raft_consensus.cc:1275] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 11 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20251026 14:59:48.446261 25599 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:48.447196 25559 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:48.447381 25637 log.cc:927] Time spent T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.087s user 0.002s sys 0.000s
I20251026 14:59:48.447685 22656 raft_consensus.cc:2410] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20251026 14:59:48.428663 25618 raft_consensus.cc:697] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [term 8 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:48.420991 22773 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:48.427717 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 44441.8us
I20251026 14:59:48.385537 25544 raft_consensus.cc:2749] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:48.438179 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:48.389387 25444 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:48.432075 25401 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.468828 22652 raft_consensus.cc:1240] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20251026 14:59:48.442092 25620 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:48.426760 25390 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:48.432675 25398 raft_consensus.cc:2749] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251026 14:59:48.490598 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 69728.8us
I20251026 14:59:48.490887 22655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1380) took 1094 ms. Trace:
I20251026 14:59:48.442521 25623 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:48.443130 25592 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.471958 22731 raft_consensus.cc:1275] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 10 index: 4. (index mismatch)
W20251026 14:59:48.484978 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 178102us
I20251026 14:59:48.432900 25193 raft_consensus.cc:697] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [term 8 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:48.470314 25570 raft_consensus.cc:2804] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:48.395067 25481 consensus_queue.cc:1048] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20251026 14:59:48.491425 22655 rpcz_store.cc:276] 1026 14:59:47.396225 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:48.061487 (+665262us) service_pool.cc:225] Handling call
1026 14:59:48.061948 (+ 461us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:48.155289 (+ 93341us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:48.155323 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:48.155937 (+ 614us) log.cc:844] Serialized 26 byte log entry
1026 14:59:48.235055 (+ 79118us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:48.235087 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:48.235150 (+ 63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:48.364677 (+129527us) spinlock_profiling.cc:243] Waited 3.42 ms on lock 0x7b440002304c. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8cdfc0905 00007fd8cdfc1322 00007fd8d0d0a231 00007fd8d0d05b5a 00007fd8d0cf991e 00007fd8d0cf8c66 00007fd8d0cf8888 00007fd8d0d022c1 00007fd8d0d02279 00007fd8d0d02209
1026 14:59:48.454111 (+ 89434us) raft_consensus.cc:1685] finished
1026 14:59:48.454144 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:48.490798 (+ 36654us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":92818,"raft.queue_time_us":219428,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":14624,"spinlock_wait_cycles":7861120,"thread_start_us":78655,"threads_started":1,"wal-append.queue_time_us":203608}
I20251026 14:59:48.434563 25627 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.485491 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:48.503307 22774 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:48.452433 25618 consensus_queue.cc:237] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.417524 25241 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:48.532768 25241 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:48.455590 25196 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:48.534938 25241 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:48.535984 25196 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.538727 25196 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:48.540180 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:48.521873 22725 raft_consensus.cc:1275] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:48.523955 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:48.502758 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 12 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:48.525640 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:48.499962 25539 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:48.527776 22651 raft_consensus.cc:3055] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:48.544488 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:48.516021 25615 raft_consensus.cc:2804] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader election won for term 10
I20251026 14:59:48.544677 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:48.545543 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:48.453534 25199 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:48.566349 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 63267.3us
I20251026 14:59:48.566766 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:48.568061 22774 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:48.440003 25591 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.560490 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:48.548607 22651 raft_consensus.cc:740] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:48.483485 25444 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.565371 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:48.572814 22774 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:48.498973 25390 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.527874 25398 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:48.525588 25545 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:48.527416 25534 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:48.575421 22622 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:48.416450 25516 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.591285 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:48.593189 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:48.518987 25598 consensus_queue.cc:1048] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20251026 14:59:48.590602 25196 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:48.516265 25599 consensus_queue.cc:1048] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20251026 14:59:48.418874 25636 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.549798 25570 raft_consensus.cc:697] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:48.601923 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:48.625847 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 50532.2us
I20251026 14:59:48.590087 23399 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:48.541040 25544 consensus_queue.cc:1048] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20251026 14:59:48.535548 25559 consensus_queue.cc:1048] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20251026 14:59:48.602010 22651 consensus_queue.cc:260] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:48.621817 22727 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:48.496989 25620 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.599633 25615 raft_consensus.cc:697] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 10 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:48.601033 25369 raft_consensus.cc:2749] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251026 14:59:48.549497 25255 consensus_queue.cc:1059] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.074s
I20251026 14:59:48.605695 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:48.535413 25643 consensus_queue.cc:1048] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.114s
I20251026 14:59:48.604581 25398 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.545485 25623 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.596052 25519 consensus_queue.cc:237] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.624398 25546 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:48.737625 25626 log.cc:927] Time spent T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.271s user 0.000s sys 0.002s
I20251026 14:59:48.815472 25528 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:48.694491 22651 raft_consensus.cc:3060] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:48.765432 22729 raft_consensus.cc:1240] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 8 LEADER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:48.758078 25563 consensus_queue.cc:1059] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
I20251026 14:59:48.738420 25641 consensus_queue.cc:1059] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.193s
I20251026 14:59:48.764394 22806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49652 (request call id 352) took 1359 ms. Trace:
I20251026 14:59:48.623549 25193 consensus_queue.cc:237] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:48.941663 22771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 351301us
I20251026 14:59:48.639032 25545 raft_consensus.cc:2804] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:48.621094 25534 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:49.022723 22652 raft_consensus.cc:1275] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:49.026453 22652 raft_consensus.cc:1275] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
W20251026 14:59:49.023344 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 412806us
W20251026 14:59:48.738230 25649 log.cc:927] Time spent T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20251026 14:59:48.802783 25515 raft_consensus.cc:3055] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 LEADER]: Stepping down as leader of term 5
W20251026 14:59:48.759078 25642 log.cc:927] Time spent T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20251026 14:59:49.037766 22731 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:48.824168 25643 consensus_queue.cc:1243] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer 562bb1e1c8e445b8baecd29c3669658e log is divergent from this leader: its last log entry 8.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20251026 14:59:48.763535 25510 raft_consensus.cc:3060] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:48.759397 25636 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:49.007725 22806 rpcz_store.cc:276] 1026 14:59:47.404808 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:47.404951 (+ 143us) service_pool.cc:225] Handling call
1026 14:59:47.405451 (+ 500us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:48.058253 (+652802us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:48.058289 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:48.149403 (+ 91114us) log.cc:844] Serialized 26 byte log entry
1026 14:59:48.525818 (+376415us) raft_consensus.cc:1645] Marking committed up to 6
1026 14:59:48.525854 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:48.525932 (+ 78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:48.764110 (+238178us) raft_consensus.cc:1685] finished
1026 14:59:48.764140 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:48.764315 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16771,"raft.queue_time_us":26406,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":4350,"thread_start_us":933206,"threads_started":2,"wal-append.queue_time_us":573700}
I20251026 14:59:48.757923 25395 raft_consensus.cc:3055] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:48.739323 25212 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:48.638370 25196 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.731885 25638 raft_consensus.cc:2749] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251026 14:59:48.738080 25620 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:48.694365 25494 consensus_queue.cc:1059] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.579s
I20251026 14:59:49.045990 25599 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:48.737150 25569 consensus_queue.cc:1059] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.551s
I20251026 14:59:48.759641 25581 consensus_queue.cc:1048] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.317s
I20251026 14:59:48.738601 25646 consensus_queue.cc:1059] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.512s
I20251026 14:59:49.108089 25645 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:49.108662 25645 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:49.110747 25645 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:49.111222 25614 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:48.757956 25570 consensus_queue.cc:237] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:48.760479 25648 consensus_queue.cc:1059] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.331s
I20251026 14:59:49.035691 25369 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:49.100301 25636 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:49.065969 25627 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:49.042583 25515 raft_consensus.cc:740] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:48.764899 25651 consensus_queue.cc:1059] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.387s
I20251026 14:59:49.072551 25395 raft_consensus.cc:740] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:59:49.104100 22801 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1314) took 2742 ms (client timeout 2048 ms). Trace:
I20251026 14:59:49.131397 22651 raft_consensus.cc:1275] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20251026 14:59:48.737344 25621 consensus_queue.cc:1059] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.561s
I20251026 14:59:49.138000 25632 raft_consensus.cc:2804] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:49.138526 25634 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:49.101507 25390 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:48.968211 25528 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:49.142010 22799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1446) took 3155 ms. Trace:
W20251026 14:59:49.149314 22801 rpcz_store.cc:269] 1026 14:59:46.357468 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:46.357626 (+ 158us) service_pool.cc:225] Handling call
1026 14:59:49.099928 (+2742302us) spinlock_profiling.cc:243] Waited 465 ms on lock 0x7b540020bcd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:49.100109 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1069992320}
W20251026 14:59:49.158635 25653 log.cc:927] Time spent T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.121s user 0.000s sys 0.000s
I20251026 14:59:49.163226 25515 consensus_queue.cc:260] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:49.167819 25634 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:49.102087 25212 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:49.169870 25395 consensus_queue.cc:260] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:49.106413 25599 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:49.170073 25623 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:49.171559 22799 rpcz_store.cc:276] 1026 14:59:46.003057 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:46.009814 (+ 6757us) service_pool.cc:225] Handling call
1026 14:59:46.024062 (+ 14248us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:46.032618 (+ 8556us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:46.032646 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:46.038883 (+ 6237us) log.cc:844] Serialized 52 byte log entry
1026 14:59:47.785418 (+1746535us) spinlock_profiling.cc:243] Waited 18.4 ms on lock 0x7b18003fe7a0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce0942f7 00007fd8d6e6d219 00007fd8ce0b0c4b 00007fd8ce0aeeef 00007fd8ce0b11b2 00007fd8d0cf82bb 00007fd8d0cfddc3 00007fd8d0cfe171 00007fd8d111965f
1026 14:59:48.045692 (+260274us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:48.045754 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:48.045860 (+ 106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:49.140830 (+1094970us) raft_consensus.cc:1685] finished
1026 14:59:49.140863 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:49.141921 (+ 1058us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":207,"raft.queue_time_us":32389,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":14194,"spinlock_wait_cycles":42324224,"thread_start_us":259997,"threads_started":1,"wal-append.queue_time_us":2322212}
I20251026 14:59:49.183873 25196 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:49.111737 25614 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:49.186368 25528 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:49.188184 25563 raft_consensus.cc:3055] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:49.189229 25647 raft_consensus.cc:3055] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:49.143780 25636 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:49.190289 25646 raft_consensus.cc:3055] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:49.193787 25515 raft_consensus.cc:3060] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:49.172735 25632 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:49.139755 25615 consensus_queue.cc:237] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:49.038298 25545 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:49.224730 25395 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:49.230790 25592 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:49.234920 22802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1348) took 1295 ms. Trace:
I20251026 14:59:49.152343 25516 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:49.143589 25369 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:49.248281 22802 rpcz_store.cc:276] 1026 14:59:47.939188 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:47.939352 (+ 164us) service_pool.cc:225] Handling call
1026 14:59:49.234592 (+1295240us) spinlock_profiling.cc:243] Waited 441 ms on lock 0x7b540004d8d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:49.234800 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1014437248}
I20251026 14:59:49.252460 22731 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 9.
W20251026 14:59:49.254087 22774 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.663s user 0.021s sys 0.024s
I20251026 14:59:49.255561 25599 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:49.256064 22774 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.665s user 0.021s sys 0.024s
W20251026 14:59:49.256793 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 684024us
I20251026 14:59:49.258164 22774 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:49.239391 25563 raft_consensus.cc:740] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:49.242798 25398 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:49.251245 25348 consensus_meta.cc:287] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.922s user 0.009s sys 0.004s
W20251026 14:59:49.252005 22772 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.009s sys 0.034s
W20251026 14:59:49.264930 22772 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.543s user 0.009s sys 0.034s
W20251026 14:59:49.261025 22773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.044s sys 0.044s
I20251026 14:59:49.247838 25632 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:49.243979 25646 raft_consensus.cc:740] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:49.248330 25545 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:49.246258 25647 raft_consensus.cc:740] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:59:49.260212 22622 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.621s user 0.008s sys 0.036s
W20251026 14:59:49.271744 22728 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.853s user 0.018s sys 0.029s
I20251026 14:59:49.253860 25648 consensus_queue.cc:1243] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer 0d684445287b450a8f738b6ad4b59821 log is divergent from this leader: its last log entry 11.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20251026 14:59:49.271512 22622 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.633s user 0.008s sys 0.036s
W20251026 14:59:49.266949 22773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.696s user 0.044s sys 0.045s
W20251026 14:59:49.274186 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:49.260449 22623 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.974s user 0.016s sys 0.029s
I20251026 14:59:49.254014 25634 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
W20251026 14:59:49.278115 22728 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.860s user 0.018s sys 0.029s
W20251026 14:59:49.273978 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 635103us
W20251026 14:59:49.274540 22655 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.009s sys 0.031s
W20251026 14:59:49.280386 22655 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.582s user 0.009s sys 0.031s
I20251026 14:59:49.269152 25348 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:49.302412 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 44345.9us
I20251026 14:59:49.305418 25647 consensus_queue.cc:260] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:49.306864 25647 raft_consensus.cc:3060] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:49.306965 25524 raft_consensus.cc:2804] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20251026 14:59:49.307613 25524 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:49.308077 25524 raft_consensus.cc:3060] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:49.300999 25255 consensus_queue.cc:1059] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20251026 14:59:49.305099 25563 consensus_queue.cc:260] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:49.315058 25563 raft_consensus.cc:3060] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:49.324621 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47996 (request call id 181) took 1272 ms. Trace:
I20251026 14:59:49.325088 22732 rpcz_store.cc:276] 1026 14:59:48.052176 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:48.052321 (+ 145us) service_pool.cc:225] Handling call
1026 14:59:48.078706 (+ 26385us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:48.313983 (+235277us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:49.324367 (+1010384us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b540006b154. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:49.324556 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":537599872}
I20251026 14:59:49.328847 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47996 (request call id 179) took 1385 ms. Trace:
I20251026 14:59:49.329468 22734 rpcz_store.cc:276] 1026 14:59:47.943866 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:47.944271 (+ 405us) service_pool.cc:225] Handling call
1026 14:59:48.021634 (+ 77363us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:48.268613 (+246979us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:49.328515 (+1059902us) spinlock_profiling.cc:243] Waited 220 ms on lock 0x7b54001ea5d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:49.328764 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":504974592}
I20251026 14:59:49.330859 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:49.335786 22734 raft_consensus.cc:2468] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:49.337227 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:49.338440 22732 raft_consensus.cc:2368] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 9. Current term is 10.
I20251026 14:59:49.340637 23400 raft_consensus.cc:1240] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:49.341614 25212 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:49.342334 22656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1634) took 1433 ms. Trace:
I20251026 14:59:49.342806 22656 rpcz_store.cc:276] 1026 14:59:47.878399 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:47.900509 (+ 22110us) service_pool.cc:225] Handling call
1026 14:59:49.311716 (+1411207us) spinlock_profiling.cc:243] Waited 536 ms on lock 0x7b54001fc554. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:49.311868 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1232861056}
I20251026 14:59:49.344996 25651 consensus_queue.cc:1059] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251026 14:59:49.346186 25614 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:49.346475 25636 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:49.347506 25369 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:49.349362 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 59689.7us
I20251026 14:59:49.348235 25646 consensus_queue.cc:260] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:49.351816 25574 log.cc:927] Time spent T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 2.651s user 0.006s sys 0.000s
I20251026 14:59:49.354174 25530 raft_consensus.cc:3055] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:49.354606 25530 raft_consensus.cc:740] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:49.356845 22802 raft_consensus.cc:1240] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 8. Current term is 9. Ops: []
I20251026 14:59:49.357030 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 12 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:49.358837 22801 raft_consensus.cc:1240] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 8. Current term is 9. Ops: []
I20251026 14:59:49.360453 22801 raft_consensus.cc:1275] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251026 14:59:49.360715 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:49.361384 22806 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:49.362262 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:49.363242 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:49.363842 22807 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:49.365191 22800 raft_consensus.cc:3060] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:49.300945 25559 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:49.365773 25559 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:49.367782 25559 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:49.368896 25591 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:49.369673 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:49.355454 25530 consensus_queue.cc:260] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:49.373557 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:49.377748 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:49.378613 22807 raft_consensus.cc:2393] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 6: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:49.362367 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:49.338049 22774 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.2.1:46185 timed out after 1.631s (SENT) [suppressed 13 similar messages]
I20251026 14:59:49.379292 22804 raft_consensus.cc:2468] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:49.311662 25348 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:49.381652 22806 raft_consensus.cc:1240] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 8. Current term is 9. Ops: [7.5-8.6]
I20251026 14:59:49.385149 22801 raft_consensus.cc:1240] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [term 10 LEADER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 9. Current term is 10. Ops: []
I20251026 14:59:49.385980 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:49.357714 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:49.357760 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:49.391847 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:49.392894 22800 raft_consensus.cc:1275] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20251026 14:59:49.395943 22732 raft_consensus.cc:1240] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 8. Current term is 9. Ops: []
I20251026 14:59:49.392983 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:49.383800 23400 raft_consensus.cc:2468] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
W20251026 14:59:49.386447 25587 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.845s user 0.017s sys 0.045s
I20251026 14:59:49.399166 22656 raft_consensus.cc:3055] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 LEADER]: Stepping down as leader of term 10
W20251026 14:59:49.399667 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:49.407335 22656 raft_consensus.cc:740] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:49.409220 22726 raft_consensus.cc:1240] T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e [term 9 LEADER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 8. Current term is 9. Ops: []
I20251026 14:59:49.402714 22734 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
W20251026 14:59:49.275102 22623 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.993s user 0.017s sys 0.029s
I20251026 14:59:49.410941 22726 raft_consensus.cc:1275] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20251026 14:59:49.411787 22734 raft_consensus.cc:1240] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:49.415177 22726 raft_consensus.cc:1240] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20251026 14:59:49.416800 22726 raft_consensus.cc:1240] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 9. Current term is 10. Ops: []
I20251026 14:59:49.419960 22734 raft_consensus.cc:1240] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 11 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 10. Current term is 11. Ops: []
W20251026 14:59:49.421298 23367 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.767s user 0.017s sys 0.041s
W20251026 14:59:49.421802 23367 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.778s user 0.018s sys 0.041s
W20251026 14:59:49.428534 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:49.429136 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:49.407266 25587 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.890s user 0.018s sys 0.046s
I20251026 14:59:49.435506 22732 raft_consensus.cc:1240] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:49.436193 22622 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:49.439066 22622 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:49.408151 22656 consensus_queue.cc:260] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:49.441212 22656 raft_consensus.cc:3060] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Advancing to term 11
I20251026 14:59:49.440357 25624 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:49.446377 22801 raft_consensus.cc:1240] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:49.441203 22622 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:49.449606 22622 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:49.435505 25586 consensus_queue.cc:1059] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20251026 14:59:49.432139 25436 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:49.438964 25632 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:49.454553 25436 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:49.454680 25658 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:49.456756 22732 raft_consensus.cc:1275] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:49.457248 22730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1643) took 1079 ms. Trace:
I20251026 14:59:49.461833 22730 rpcz_store.cc:276] 1026 14:59:48.378255 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:48.425545 (+ 47290us) service_pool.cc:225] Handling call
1026 14:59:48.467291 (+ 41746us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:48.522600 (+ 55309us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:48.522649 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:48.522694 (+ 45us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:48.522728 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:49.457023 (+934295us) spinlock_profiling.cc:243] Waited 55.1 ms on lock 0x7b54000b9fd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:49.457067 (+ 44us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:49.457191 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":126708608}
W20251026 14:59:49.447989 25481 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.643s user 0.030s sys 0.045s
W20251026 14:59:49.458950 22699 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.591s user 0.021s sys 0.020s
I20251026 14:59:49.446089 25634 raft_consensus.cc:2804] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20251026 14:59:49.474812 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:49.483122 22656 raft_consensus.cc:1275] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20251026 14:59:49.483826 22656 pending_rounds.cc:85] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29: Aborting all ops after (but not including) 5
I20251026 14:59:49.484177 22656 pending_rounds.cc:107] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20251026 14:59:49.459873 25588 raft_consensus.cc:2804] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:49.484670 22656 raft_consensus.cc:2889] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:49.482980 22730 raft_consensus.cc:1240] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:49.442834 25601 consensus_queue.cc:1048] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20251026 14:59:49.488801 22730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1231) took 1004 ms. Trace:
I20251026 14:59:49.489120 22730 rpcz_store.cc:276] 1026 14:59:48.484259 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:49.463410 (+979151us) service_pool.cc:225] Handling call
1026 14:59:49.482717 (+ 19307us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:49.488513 (+ 5796us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:49.488732 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:49.487200 22657 raft_consensus.cc:606] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:49.490325 22657 raft_consensus.cc:614] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251026 14:59:49.442992 25604 consensus_queue.cc:1059] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.029s
I20251026 14:59:49.498167 25632 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:49.503072 25520 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:49.475010 25634 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
W20251026 14:59:49.469707 25481 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.704s user 0.031s sys 0.046s
I20251026 14:59:49.494135 22653 raft_consensus.cc:1275] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251026 14:59:49.492314 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:49.505045 25667 raft_consensus.cc:3055] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Stepping down as leader of term 8
I20251026 14:59:49.491927 25588 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:49.525473 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 418123us
I20251026 14:59:49.533573 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:49.534998 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:49.535610 22730 raft_consensus.cc:1240] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
W20251026 14:59:49.544386 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 94876.6us
I20251026 14:59:49.547982 25212 consensus_queue.cc:1048] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20251026 14:59:49.550199 25212 consensus_queue.cc:1048] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20251026 14:59:49.535800 22656 raft_consensus.cc:2468] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:49.553972 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:49.456415 25624 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:49.523291 25634 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:49.559669 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:49.562199 22656 raft_consensus.cc:3060] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:49.524359 25613 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:49.524673 25667 raft_consensus.cc:740] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:49.564033 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:49.539049 25588 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:49.531157 25614 raft_consensus.cc:493] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:49.570123 22807 raft_consensus.cc:2468] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
I20251026 14:59:49.579902 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:49.580166 22622 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:49.555011 25510 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:49.556874 25601 raft_consensus.cc:2804] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:49.589725 22773 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:49.592316 22697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.626s user 0.010s sys 0.013s
W20251026 14:59:49.592898 22697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.627s user 0.010s sys 0.013s
I20251026 14:59:49.594579 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:59:49.675002 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 94952.5us
I20251026 14:59:49.560664 25436 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:49.694690 22773 leader_election.cc:400] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507) with higher term. Message: Invalid argument: T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 9. Current term is 10.
I20251026 14:59:49.469365 25658 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:49.594653 25614 raft_consensus.cc:515] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:49.713059 25664 log.cc:927] Time spent T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.158s user 0.000s sys 0.000s
I20251026 14:59:49.714179 25617 raft_consensus.cc:3055] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Stepping down as leader of term 8
I20251026 14:59:49.714365 25614 leader_election.cc:290] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:49.714658 25617 raft_consensus.cc:740] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:49.714718 25658 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:49.717021 22728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1639) took 1340 ms. Trace:
W20251026 14:59:49.719432 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:49.697033 22773 leader_election.cc:403] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20251026 14:59:49.578701 25563 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:59:49.712178 23368 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:49.578488 25505 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:49.721418 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:49.567691 25604 consensus_queue.cc:1048] T 21e9405b016a4df592291cba0e961b07 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.127s
I20251026 14:59:49.717573 22728 rpcz_store.cc:276] 1026 14:59:48.376971 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:48.377150 (+ 179us) service_pool.cc:225] Handling call
1026 14:59:48.380328 (+ 3178us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:48.382401 (+ 2073us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:48.382431 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:48.382888 (+ 457us) log.cc:844] Serialized 26 byte log entry
1026 14:59:49.279689 (+896801us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:49.279746 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:49.279843 (+ 97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:49.716739 (+436896us) raft_consensus.cc:1685] finished
1026 14:59:49.716770 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:49.716948 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1632,"raft.queue_time_us":8594,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":29973,"thread_start_us":861250,"threads_started":1,"wal-append.queue_time_us":1095048}
I20251026 14:59:49.565128 25632 consensus_queue.cc:1059] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.151s
I20251026 14:59:49.522616 25553 consensus_queue.cc:1048] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.070s
I20251026 14:59:49.729429 25668 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:49.674917 25520 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:49.734925 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 49069.7us
I20251026 14:59:49.592168 25601 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:49.728662 25635 raft_consensus.cc:2804] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251026 14:59:49.748353 22650 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:49.755725 22624 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:49.754426 22733 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.018s sys 0.013s
I20251026 14:59:49.756139 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:49.759990 22728 raft_consensus.cc:3060] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 7
I20251026 14:59:49.676751 25510 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:49.744369 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:49.578352 25613 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:49.731909 25545 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:49.715623 25617 consensus_queue.cc:260] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:49.760648 25528 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:49.954909 22625 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.698s user 0.006s sys 0.026s
W20251026 14:59:50.005767 22733 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.764s user 0.024s sys 0.019s
I20251026 14:59:49.832229 22697 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:49.838100 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:49.786936 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:49.822403 22656 raft_consensus.cc:2468] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
W20251026 14:59:49.830605 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 74971.2us
W20251026 14:59:49.831547 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:49.861728 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 523631us
W20251026 14:59:50.016840 22625 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.941s user 0.006s sys 0.038s
W20251026 14:59:50.015175 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 183056us
I20251026 14:59:50.016209 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:49.810119 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:50.019060 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:50.016686 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:49.733006 25666 raft_consensus.cc:493] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
W20251026 14:59:49.804437 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:50.019835 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:49.742410 25655 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.013s sys 0.014s
W20251026 14:59:49.758718 25519 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.009s sys 0.020s
I20251026 14:59:49.750636 25658 consensus_queue.cc:1059] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.278s
I20251026 14:59:49.748369 25505 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:50.085233 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:50.163254 22804 raft_consensus.cc:2393] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 8: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:50.252808 22652 raft_consensus.cc:3055] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 14:59:50.265419 22652 raft_consensus.cc:740] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:50.000263 25663 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 10
W20251026 14:59:49.905885 25570 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.668s user 0.002s sys 0.017s
I20251026 14:59:50.161000 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:50.023578 22728 raft_consensus.cc:2468] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:50.000036 25674 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:49.955651 25405 raft_consensus.cc:2804] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251026 14:59:50.263715 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:50.006302 25587 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.020s sys 0.032s
I20251026 14:59:50.027160 25591 raft_consensus.cc:2749] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251026 14:59:50.084148 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:50.132016 22697 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:50.007692 25669 log.cc:927] Time spent T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.424s user 0.000s sys 0.001s
I20251026 14:59:49.759958 25390 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:50.287719 25524 consensus_meta.cc:287] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.979s user 0.009s sys 0.001s
I20251026 14:59:50.021772 25665 consensus_queue.cc:1059] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.286s
I20251026 14:59:49.734017 25563 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:50.266423 22652 consensus_queue.cc:260] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:49.761879 25601 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
W20251026 14:59:50.244427 23368 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:49.957086 25193 raft_consensus.cc:2804] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:49.801052 25635 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:49.744484 23399 raft_consensus.cc:3055] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 8 LEADER]: Stepping down as leader of term 8
W20251026 14:59:50.265843 25519 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.038s user 0.012s sys 0.036s
I20251026 14:59:49.946972 25592 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:50.277837 23403 raft_consensus.cc:2393] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 7: Already voted for candidate 248493ec312e4398b828c6cb00e5c4b1 in this term.
I20251026 14:59:50.359627 25635 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Advancing to term 10
W20251026 14:59:50.365142 25679 log.cc:927] Time spent T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.088s user 0.000s sys 0.001s
W20251026 14:59:50.366318 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 102932us
W20251026 14:59:50.245671 25656 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.089s user 0.005s sys 0.030s
I20251026 14:59:49.577979 25667 consensus_queue.cc:260] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.312853 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:50.277920 25570 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.041s user 0.008s sys 0.023s
I20251026 14:59:49.920799 25614 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:59:50.317740 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:50.345245 25510 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
W20251026 14:59:50.386075 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 345839us
I20251026 14:59:50.022437 25199 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:50.022138 25546 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:50.022259 25615 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:50.202579 25624 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:50.202232 25645 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:49.774122 25494 consensus_queue.cc:1048] T aceef9b41bab4ad3bf67b4d597a7fe11 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.540s
I20251026 14:59:50.153266 25581 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:59:50.294816 25655 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.055s user 0.029s sys 0.033s
I20251026 14:59:50.277330 25481 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:49.759593 25520 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:50.312518 25587 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.845s user 0.035s sys 0.054s
I20251026 14:59:50.341236 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:50.302822 25666 raft_consensus.cc:515] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:50.344739 25563 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:50.344794 25505 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:49.809939 25668 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.298627 25658 consensus_queue.cc:1243] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Peer 562bb1e1c8e445b8baecd29c3669658e log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20251026 14:59:50.298959 25405 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:50.345966 22652 raft_consensus.cc:3060] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:50.345623 22656 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 6.
I20251026 14:59:50.027691 25528 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.355161 22733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1638) took 1978 ms. Trace:
I20251026 14:59:50.326712 25444 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:50.153684 25675 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:59:50.378921 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:50.380139 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:50.489537 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:50.489892 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:50.490155 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:50.022567 25672 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:50.083716 25212 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:50.433789 22799 raft_consensus.cc:1240] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:50.438791 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:50.339164 25390 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.349296 25524 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:50.359858 23399 raft_consensus.cc:740] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:50.377868 25591 raft_consensus.cc:2749] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20251026 14:59:50.388368 25656 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.647s user 0.011s sys 0.030s
I20251026 14:59:50.375803 25592 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.443959 25398 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:50.469159 25405 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:50.447698 25645 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.458300 25561 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:50.506389 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:50.457587 25604 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:50.430016 25614 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.434233 25546 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.458592 25481 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.460387 25519 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:50.465358 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:50.381115 25523 consensus_queue.cc:1048] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.083s
I20251026 14:59:50.440464 25615 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.469722 22733 rpcz_store.cc:276] 1026 14:59:48.376656 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:48.380654 (+ 3998us) service_pool.cc:225] Handling call
1026 14:59:48.380951 (+ 297us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:48.424434 (+ 43483us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:48.424468 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:48.428775 (+ 4307us) log.cc:844] Serialized 26 byte log entry
1026 14:59:49.241229 (+812454us) spinlock_profiling.cc:243] Waited 2.67 ms on lock 0x7b18003343c0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce0942f7 00007fd8d6e6d219 00007fd8ce0b0c4b 00007fd8ce0aeeef 00007fd8ce0b11b2 00007fd8d0cf82bb 00007fd8d0cfddc3 00007fd8d0cfe171 00007fd8d111965f
1026 14:59:50.014659 (+773430us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:50.014698 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:50.014775 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:50.348044 (+333269us) raft_consensus.cc:1685] finished
1026 14:59:50.348079 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:50.355066 (+ 6987us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31435,"raft.queue_time_us":43712,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":8950,"spinlock_wait_cycles":6143360,"thread_start_us":773024,"threads_started":1,"wal-append.queue_time_us":1901314}
W20251026 14:59:50.568965 25695 log.cc:927] Time spent T 5b0536aab3b541a2a846d187e111c09c P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.102s user 0.000s sys 0.001s
I20251026 14:59:50.569628 25592 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:50.558416 25398 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.340706 25674 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:50.512889 25688 log.cc:927] Time spent T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.191s user 0.000s sys 0.001s
I20251026 14:59:50.566440 22806 raft_consensus.cc:1240] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 4. Current term is 5. Ops: []
I20251026 14:59:50.568885 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:50.504284 25395 consensus_meta.cc:287] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 1.253s user 0.013s sys 0.000s
I20251026 14:59:50.509583 25505 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:50.572432 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:50.572731 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1647) took 2041 ms. Trace:
I20251026 14:59:50.576048 22652 raft_consensus.cc:1275] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20251026 14:59:50.576857 22732 rpcz_store.cc:276] 1026 14:59:48.531511 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:49.465916 (+934405us) service_pool.cc:225] Handling call
1026 14:59:49.466326 (+ 410us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:49.471034 (+ 4708us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:49.471080 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:49.497387 (+ 26307us) log.cc:844] Serialized 52 byte log entry
1026 14:59:49.906362 (+408975us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:49.906417 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:49.906505 (+ 88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:50.572523 (+666018us) raft_consensus.cc:1685] finished
1026 14:59:50.572542 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:50.572679 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4377,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":105,"thread_start_us":406897,"threads_started":1,"wal-append.queue_time_us":949167}
I20251026 14:59:50.579214 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:50.579972 22732 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:50.573257 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1228) took 2089 ms. Trace:
I20251026 14:59:50.581763 22734 rpcz_store.cc:276] 1026 14:59:48.483454 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:49.424863 (+941409us) service_pool.cc:225] Handling call
1026 14:59:49.425233 (+ 370us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:49.425560 (+ 327us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:49.425599 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:49.426199 (+ 600us) log.cc:844] Serialized 26 byte log entry
1026 14:59:49.728773 (+302574us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:49.728839 (+ 66us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:49.728939 (+ 100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:50.573020 (+844081us) raft_consensus.cc:1685] finished
1026 14:59:50.573039 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:50.573188 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":288311,"threads_started":1,"wal-append.queue_time_us":1072712}
I20251026 14:59:50.585932 22728 raft_consensus.cc:2393] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 7: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:50.501186 25563 consensus_queue.cc:1059] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.073s
I20251026 14:59:50.586133 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:50.587334 22728 raft_consensus.cc:1240] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:50.577296 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1394) took 2047 ms. Trace:
I20251026 14:59:50.531576 25587 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:50.593189 22652 rpcz_store.cc:276] 1026 14:59:48.529985 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:49.041693 (+511708us) service_pool.cc:225] Handling call
1026 14:59:49.042072 (+ 379us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:50.245239 (+1203167us) spinlock_profiling.cc:243] Waited 6.59 ms on lock 0x7b5400125d20. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d10d5d10 00007fd8d115fd50 00007fd8d1160dc1 00007fd8d115d92f 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c
1026 14:59:50.577020 (+331781us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:50.577223 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15149440}
W20251026 14:59:50.572223 25669 log.cc:927] Time spent T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20251026 14:59:50.596076 22730 raft_consensus.cc:2410] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20251026 14:59:50.509347 25212 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.596740 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:50.605644 22733 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:50.617978 22652 raft_consensus.cc:3055] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:50.571830 25545 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:50.455574 25581 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.512224 25672 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.628032 22652 raft_consensus.cc:740] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:50.634328 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1227) took 2151 ms. Trace:
I20251026 14:59:50.640595 22732 raft_consensus.cc:1275] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:50.642335 22652 consensus_queue.cc:260] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.629671 25506 raft_consensus.cc:2749] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20251026 14:59:50.644971 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1636) took 1253 ms. Trace:
I20251026 14:59:50.641204 22726 rpcz_store.cc:276] 1026 14:59:48.483133 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:49.419196 (+936063us) service_pool.cc:225] Handling call
1026 14:59:49.419442 (+ 246us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:49.430833 (+ 11391us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:49.430884 (+ 51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:49.437744 (+ 6860us) log.cc:844] Serialized 26 byte log entry
1026 14:59:50.005916 (+568172us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:50.005957 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:50.006028 (+ 71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:50.633963 (+627935us) raft_consensus.cc:1685] finished
1026 14:59:50.633998 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:50.634246 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1243,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":3696,"thread_start_us":385189,"threads_started":1,"wal-append.queue_time_us":1065424}
I20251026 14:59:50.647089 22726 raft_consensus.cc:1240] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [term 11 LEADER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20251026 14:59:50.645401 22659 rpcz_store.cc:276] 1026 14:59:49.391952 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:49.393465 (+ 1513us) service_pool.cc:225] Handling call
1026 14:59:49.393836 (+ 371us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:49.394282 (+ 446us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:49.394324 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:49.395088 (+ 764us) log.cc:844] Serialized 26 byte log entry
1026 14:59:49.943327 (+548239us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:50.265548 (+322221us) log.cc:844] Serialized 14 byte log entry
1026 14:59:50.626166 (+360618us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:50.626251 (+ 85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:50.644711 (+ 18460us) raft_consensus.cc:1685] finished
1026 14:59:50.644743 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:50.644906 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14362,"prepare.queue_time_us":57137,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":65,"raft.queue_time_us":116863,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":4693,"thread_start_us":1040456,"threads_started":3,"wal-append.queue_time_us":761929}
I20251026 14:59:50.636636 23403 raft_consensus.cc:2393] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 7: Already voted for candidate 248493ec312e4398b828c6cb00e5c4b1 in this term.
I20251026 14:59:50.572963 25604 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:50.635046 25698 log.cc:927] Time spent T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.288s user 0.001s sys 0.000s
I20251026 14:59:50.653891 22726 raft_consensus.cc:1240] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [term 8 LEADER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:50.658797 22732 raft_consensus.cc:1217] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 11 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[5.3-11.6] Dedup: 5.3->[7.4-11.6]
I20251026 14:59:50.576594 25519 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.589112 25505 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.661762 22732 pending_rounds.cc:85] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e: Aborting all ops after (but not including) 3
I20251026 14:59:50.664759 22732 pending_rounds.cc:107] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20251026 14:59:50.665199 22732 raft_consensus.cc:2889] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:50.663645 22726 raft_consensus.cc:1275] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251026 14:59:50.635040 25663 raft_consensus.cc:2749] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507) with higher term. Message: Invalid argument: T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for earlier term 9. Current term is 10.
I20251026 14:59:50.637935 25510 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:50.587198 25561 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:50.626717 25587 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.668856 22652 raft_consensus.cc:3060] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:50.503626 25675 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.676662 25675 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:50.678470 22808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1452) took 4055 ms. Trace:
I20251026 14:59:50.678701 25561 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:50.678930 22808 rpcz_store.cc:276] 1026 14:59:46.623439 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:46.912468 (+289029us) service_pool.cc:225] Handling call
1026 14:59:46.912872 (+ 404us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:47.883231 (+970359us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:50.678202 (+2794971us) spinlock_profiling.cc:243] Waited 969 ms on lock 0x7b54001ac554. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:50.678393 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2229413632}
I20251026 14:59:50.680428 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:50.679453 25697 log.cc:927] Time spent T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20251026 14:59:50.681180 25587 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:50.683535 25588 consensus_meta.cc:287] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 1.089s user 0.012s sys 0.000s
I20251026 14:59:50.683653 25674 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:50.683465 25510 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.686208 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:50.684134 25588 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.688619 25510 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:50.688974 22654 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:50.689114 25588 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:50.693428 22653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1622) took 2900 ms. Trace:
I20251026 14:59:50.689915 23404 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
W20251026 14:59:50.701050 25612 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20251026 14:59:50.701545 25614 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:50.693984 22653 rpcz_store.cc:276] 1026 14:59:47.785192 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:48.166155 (+380963us) service_pool.cc:225] Handling call
1026 14:59:48.189677 (+ 23522us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:49.522847 (+1333170us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:50.685014 (+1162167us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b540034c454. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:50.685182 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2992278400}
I20251026 14:59:50.703598 22801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1477) took 1141 ms. Trace:
I20251026 14:59:50.702392 25212 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:50.702677 25481 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:50.685352 25704 raft_consensus.cc:3055] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 LEADER]: Stepping down as leader of term 9
I20251026 14:59:50.671833 25398 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:50.719460 22733 raft_consensus.cc:2468] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 9.
I20251026 14:59:50.707852 22801 rpcz_store.cc:276] 1026 14:59:49.562386 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:49.562548 (+ 162us) service_pool.cc:225] Handling call
1026 14:59:49.562876 (+ 328us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:49.563221 (+ 345us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:49.563250 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:49.565170 (+ 1920us) log.cc:844] Serialized 26 byte log entry
1026 14:59:50.005773 (+440603us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:50.005839 (+ 66us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:50.005910 (+ 71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:50.703386 (+697476us) raft_consensus.cc:1685] finished
1026 14:59:50.703405 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:50.703537 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":622,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":8368,"thread_start_us":368484,"threads_started":1,"wal-append.queue_time_us":761122}
I20251026 14:59:50.721117 25524 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:50.670966 25545 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:50.626420 23399 consensus_queue.cc:260] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:50.740846 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:50.760569 22801 raft_consensus.cc:2468] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:50.731896 25704 raft_consensus.cc:740] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:50.721042 25668 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:50.743639 25621 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:50.745700 25618 raft_consensus.cc:3055] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:50.748172 25646 raft_consensus.cc:3060] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
W20251026 14:59:50.749699 25627 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 11 current term: 11 is higher.
I20251026 14:59:50.750444 22806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49652 (request call id 358) took 1327 ms. Trace:
W20251026 14:59:50.760704 25634 consensus_meta.cc:287] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 1.192s user 0.002s sys 0.010s
I20251026 14:59:50.733016 25709 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:50.728677 25520 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:50.770776 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 744284us
W20251026 14:59:50.784056 23369 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:50.817176 22697 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:50.819159 25581 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:50.760118 25528 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:50.774546 25618 raft_consensus.cc:740] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:50.770641 25666 leader_election.cc:290] T 5373ef47363d446fbf9d697ed507e754 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:50.789680 25621 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.831604 25517 raft_consensus.cc:3055] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Stepping down as leader of term 8
I20251026 14:59:50.834219 25517 raft_consensus.cc:740] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:50.842061 22652 raft_consensus.cc:1275] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20251026 14:59:50.820062 25635 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:50.802587 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:50.804162 22806 rpcz_store.cc:276] 1026 14:59:49.423520 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:49.423718 (+ 198us) service_pool.cc:225] Handling call
1026 14:59:49.424097 (+ 379us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:50.574905 (+1150808us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:50.750093 (+175188us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54001b8ad4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:50.750352 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2608987008}
I20251026 14:59:50.850029 22801 raft_consensus.cc:3060] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:50.848528 25517 consensus_queue.cc:260] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.850507 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 12 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:50.850870 25517 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:50.851768 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:50.852794 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:50.751828 25694 log.cc:927] Time spent T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.303s user 0.000s sys 0.001s
I20251026 14:59:50.805662 25615 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:50.805351 25601 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.763074 25545 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.800092 22654 raft_consensus.cc:1275] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 10 index: 4. (term mismatch)
I20251026 14:59:50.760641 25390 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:50.845883 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:50.828145 25481 raft_consensus.cc:2804] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:50.854012 22697 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:50.765005 25704 consensus_queue.cc:260] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.832310 25645 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:50.815865 25634 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.812101 22733 raft_consensus.cc:1240] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:50.827977 25709 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:50.940235 22624 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.007s sys 0.036s
I20251026 14:59:50.957511 22808 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 10.
I20251026 14:59:50.844645 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1639) took 1444 ms. Trace:
I20251026 14:59:51.323453 22733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1241) took 1483 ms. Trace:
W20251026 14:59:51.096210 22807 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.006s sys 0.004s
I20251026 14:59:51.323900 22733 rpcz_store.cc:276] 1026 14:59:49.840423 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:50.740016 (+899593us) service_pool.cc:225] Handling call
1026 14:59:50.811862 (+ 71846us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:51.323179 (+511317us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:51.323393 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:51.324519 22807 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.859s user 0.008s sys 0.010s
W20251026 14:59:50.939847 22773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.023s sys 0.019s
W20251026 14:59:51.325219 22773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.938s user 0.034s sys 0.039s
I20251026 14:59:51.325512 22654 pending_rounds.cc:85] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29: Aborting all ops after (but not including) 3
I20251026 14:59:51.325871 22654 pending_rounds.cc:107] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29: Aborting uncommitted NO_OP operation due to leader change: 9.4
I20251026 14:59:51.323467 22652 rpcz_store.cc:276] 1026 14:59:49.400389 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:50.612357 (+1211968us) service_pool.cc:225] Handling call
1026 14:59:50.612824 (+ 467us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:50.842634 (+229810us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:50.844565 (+ 1931us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:51.322836 22624 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.935s user 0.014s sys 0.059s
I20251026 14:59:51.326411 22654 raft_consensus.cc:2889] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251026 14:59:51.327040 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 939317us
W20251026 14:59:51.072597 22625 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.018s sys 0.025s
W20251026 14:59:51.328404 22625 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.945s user 0.024s sys 0.042s
I20251026 14:59:51.329504 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:51.330163 22652 raft_consensus.cc:3060] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
W20251026 14:59:50.892624 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:51.331835 25481 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:51.332517 25481 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:50.952685 25618 consensus_queue.cc:260] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:51.333904 25618 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:51.335379 25505 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:51.336798 22802 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
I20251026 14:59:51.337586 22652 raft_consensus.cc:2468] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:51.338727 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:51.338826 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:51.342578 25481 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:50.982394 25604 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:50.945574 25534 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
W20251026 14:59:51.337872 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:50.987426 25516 raft_consensus.cc:3055] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Stepping down as leader of term 8
I20251026 14:59:51.345849 25481 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:51.347111 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:51.339600 22652 raft_consensus.cc:2410] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20251026 14:59:51.354974 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 531030us
W20251026 14:59:51.355275 22801 consensus_meta.cc:287] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.505s user 0.010s sys 0.001s
W20251026 14:59:50.997325 25657 log.cc:927] Time spent T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 1.379s user 0.001s sys 0.003s
I20251026 14:59:51.356103 22697 leader_election.cc:304] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
W20251026 14:59:51.347730 22622 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.962s user 0.007s sys 0.022s
I20251026 14:59:51.355039 22808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1363) took 1247 ms. Trace:
W20251026 14:59:51.357436 22622 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.972s user 0.007s sys 0.022s
I20251026 14:59:51.357548 22808 rpcz_store.cc:276] 1026 14:59:50.075609 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:50.682081 (+606472us) service_pool.cc:225] Handling call
1026 14:59:51.323225 (+641144us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:51.356299 22654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1632) took 3567 ms. Trace:
I20251026 14:59:51.358964 22654 rpcz_store.cc:276] 1026 14:59:47.789791 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:48.237003 (+447212us) service_pool.cc:225] Handling call
1026 14:59:48.237532 (+ 529us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:50.688740 (+2451208us) spinlock_profiling.cc:243] Waited 885 ms on lock 0x7b58006035d0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d10b9ffa 00007fd8d113c097 00007fd8d1154b87 00007fd8d115637e 00007fd8d11606d9 00007fd8d1160fa6 00007fd8d115d92f 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34
1026 14:59:51.355785 (+667045us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:51.356024 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2036497152}
I20251026 14:59:51.360165 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:51.360821 22654 raft_consensus.cc:2468] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:51.364243 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:51.344292 25709 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:51.362890 22697 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:51.379004 22697 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:51.368191 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:51.365254 22652 raft_consensus.cc:2468] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 9.
I20251026 14:59:51.380877 22808 raft_consensus.cc:2468] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:51.350621 22802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1372) took 1074 ms. Trace:
W20251026 14:59:50.961323 25683 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.614s user 0.028s sys 0.058s
I20251026 14:59:51.356070 22801 raft_consensus.cc:2468] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:51.366091 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:50.990260 25681 log.cc:927] Time spent T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.712s user 0.001s sys 0.000s
I20251026 14:59:51.366631 25711 raft_consensus.cc:2749] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:51.373735 23404 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:51.346407 25516 raft_consensus.cc:740] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:51.374625 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:51.422441 25516 consensus_queue.cc:260] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:51.424275 25516 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:51.426401 25634 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:51.427853 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1654) took 1667 ms. Trace:
W20251026 14:59:51.391305 25683 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 1.040s user 0.039s sys 0.084s
I20251026 14:59:51.375805 22773 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:51.436915 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 61207.1us
W20251026 14:59:51.376240 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:51.437992 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:50.940240 25615 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:50.855556 25546 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:51.349833 25638 raft_consensus.cc:493] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:50.982051 25621 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:51.391151 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 62745.1us
I20251026 14:59:51.390403 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 12 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:51.394829 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:51.465559 22624 leader_election.cc:304] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:51.465937 22652 raft_consensus.cc:2393] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 9: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:51.467298 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:51.405689 25635 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 election: Requested vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:51.468747 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:51.381515 22808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1374) took 1068 ms. Trace:
W20251026 14:59:51.398046 22800 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.825s user 0.012s sys 0.021s
W20251026 14:59:51.470060 22800 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.903s user 0.012s sys 0.021s
I20251026 14:59:51.469889 22808 rpcz_store.cc:276] 1026 14:59:50.312932 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.367350 (+1054418us) service_pool.cc:225] Handling call
1026 14:59:51.381442 (+ 14092us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:51.382467 22802 rpcz_store.cc:276] 1026 14:59:50.275867 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.338308 (+1062441us) service_pool.cc:225] Handling call
1026 14:59:51.350553 (+ 12245us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:51.471392 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:51.472118 22802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1375) took 1126 ms. Trace:
I20251026 14:59:51.472510 22802 rpcz_store.cc:276] 1026 14:59:50.345824 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.471147 (+1125323us) service_pool.cc:225] Handling call
1026 14:59:51.472063 (+ 916us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:51.369648 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:51.368283 25481 raft_consensus.cc:2804] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:51.358419 25613 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:51.405457 25604 consensus_queue.cc:1048] T 6d267e50950c4ef2b95c745e7103cafb P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.017s
I20251026 14:59:51.428295 22734 rpcz_store.cc:276] 1026 14:59:49.760391 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:50.584724 (+824333us) service_pool.cc:225] Handling call
1026 14:59:50.585054 (+ 330us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:50.586829 (+ 1775us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:50.586856 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:50.593801 (+ 6945us) log.cc:844] Serialized 26 byte log entry
1026 14:59:51.327840 (+734039us) raft_consensus.cc:1645] Marking committed up to 6
1026 14:59:51.327908 (+ 68us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:51.328011 (+ 103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:51.427579 (+ 99568us) raft_consensus.cc:1685] finished
1026 14:59:51.427605 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:51.427786 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":998,"spinlock_wait_cycles":16347904,"thread_start_us":223350,"threads_started":1,"wal-append.queue_time_us":743286}
I20251026 14:59:51.375027 25601 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:51.439509 23369 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:51.381395 25704 raft_consensus.cc:3060] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:51.392506 25390 raft_consensus.cc:3060] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:51.442371 25717 consensus_queue.cc:1059] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
W20251026 14:59:51.458221 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 360150us
W20251026 14:59:51.522302 22731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 1649) took 2950 ms (client timeout 2043 ms). Trace:
W20251026 14:59:51.459874 22774 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.862s user 0.016s sys 0.051s
I20251026 14:59:51.395601 22801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1378) took 1048 ms. Trace:
I20251026 14:59:51.464854 22654 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 11.
W20251026 14:59:51.522799 22731 rpcz_store.cc:269] 1026 14:59:48.532249 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:48.544448 (+ 12199us) service_pool.cc:225] Handling call
1026 14:59:51.482482 (+2938034us) spinlock_profiling.cc:243] Waited 476 ms on lock 0x7b540008ded4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:51.482599 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1095739264}
W20251026 14:59:51.523252 22774 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.925s user 0.016s sys 0.051s
I20251026 14:59:51.457711 25711 raft_consensus.cc:2804] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader election won for term 9
I20251026 14:59:51.523286 22801 rpcz_store.cc:276] 1026 14:59:50.347012 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:50.789916 (+442904us) service_pool.cc:225] Handling call
1026 14:59:51.395513 (+605597us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:51.489101 25481 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:51.479497 22658 raft_consensus.cc:1240] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 9. Ops: []
I20251026 14:59:51.479825 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:51.486966 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:51.450191 25287 raft_consensus.cc:2804] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:51.519914 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47996 (request call id 183) took 1500 ms. Trace:
W20251026 14:59:51.473798 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:51.475018 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:51.545017 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:51.544727 22734 rpcz_store.cc:276] 1026 14:59:50.019669 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.518915 (+1499246us) service_pool.cc:225] Handling call
1026 14:59:51.519300 (+ 385us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:51.519527 (+ 227us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:51.519563 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:51.519611 (+ 48us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:51.519646 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:51.519709 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:51.519843 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:51.493463 25405 consensus_meta.cc:287] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.918s user 0.009s sys 0.005s
I20251026 14:59:51.549791 25405 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:51.533236 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:51.533864 22731 raft_consensus.cc:1275] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251026 14:59:51.552482 22731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1666) took 1529 ms. Trace:
I20251026 14:59:51.552592 22654 raft_consensus.cc:2410] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20251026 14:59:51.552888 22731 rpcz_store.cc:276] 1026 14:59:50.022836 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.533180 (+1510344us) service_pool.cc:225] Handling call
1026 14:59:51.533498 (+ 318us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:51.552213 (+ 18715us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:51.552428 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:51.553809 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:51.554689 22731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1668) took 1170 ms. Trace:
I20251026 14:59:51.555195 25405 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:51.555876 22804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44594 (request call id 1479) took 1990 ms. Trace:
I20251026 14:59:51.556273 22804 rpcz_store.cc:276] 1026 14:59:49.565811 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:49.572757 (+ 6946us) service_pool.cc:225] Handling call
1026 14:59:51.555685 (+1982928us) spinlock_profiling.cc:243] Waited 568 ms on lock 0x7b54000c4c54. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:51.555821 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1305748352}
I20251026 14:59:51.558970 25506 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:51.559573 25506 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:51.560977 22733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47996 (request call id 182) took 1542 ms. Trace:
I20251026 14:59:51.561388 22733 rpcz_store.cc:276] 1026 14:59:50.018855 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.354524 (+1335669us) service_pool.cc:225] Handling call
1026 14:59:51.379465 (+ 24941us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:51.444322 (+ 64857us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:51.444357 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:51.444397 (+ 40us) raft_consensus.cc:1645] Marking committed up to 6
1026 14:59:51.444426 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:51.560737 (+116311us) spinlock_profiling.cc:243] Waited 64.6 ms on lock 0x7b540028cb54. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:51.560781 (+ 44us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:51.560918 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":148644224}
I20251026 14:59:51.488391 25621 consensus_queue.cc:1048] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.059s
I20251026 14:59:51.567724 22700 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:51.488860 25634 raft_consensus.cc:2749] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20251026 14:59:51.547389 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:51.458977 25615 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:51.549224 22802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1377) took 1202 ms. Trace:
I20251026 14:59:51.553994 25586 raft_consensus.cc:3055] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Stepping down as leader of term 8
I20251026 14:59:51.525414 22773 leader_election.cc:304] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:51.578205 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1637) took 3194 ms. Trace:
I20251026 14:59:51.568416 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:51.580628 22804 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:51.479310 22655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1381) took 2960 ms. Trace:
I20251026 14:59:51.581732 22655 rpcz_store.cc:276] 1026 14:59:48.518786 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:48.622475 (+103689us) service_pool.cc:225] Handling call
1026 14:59:48.622798 (+ 323us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:49.299483 (+676685us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:49.299517 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:49.299567 (+ 50us) raft_consensus.cc:1645] Marking committed up to 6
1026 14:59:49.299605 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:51.478989 (+2179384us) spinlock_profiling.cc:243] Waited 75 ms on lock 0x7b54001d92d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:51.479041 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:51.479235 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":62638,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":89,"spinlock_wait_cycles":172444416,"thread_start_us":583092,"threads_started":1}
W20251026 14:59:51.569336 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:51.571110 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:51.571341 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:51.585068 22808 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
I20251026 14:59:51.585759 22808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1383) took 1027 ms. Trace:
I20251026 14:59:51.586180 22808 rpcz_store.cc:276] 1026 14:59:50.558300 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.567054 (+1008754us) service_pool.cc:225] Handling call
1026 14:59:51.585683 (+ 18629us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:51.571792 22734 raft_consensus.cc:1275] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251026 14:59:51.573628 22801 raft_consensus.cc:2410] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20251026 14:59:51.555071 22731 rpcz_store.cc:276] 1026 14:59:50.383824 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.553786 (+1169962us) service_pool.cc:225] Handling call
1026 14:59:51.554089 (+ 303us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:51.554308 (+ 219us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:51.554346 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:51.554399 (+ 53us) raft_consensus.cc:1645] Marking committed up to 6
1026 14:59:51.554438 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:51.554514 (+ 76us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:51.554631 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:51.574220 22802 rpcz_store.cc:276] 1026 14:59:50.346656 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.473262 (+1126606us) service_pool.cc:225] Handling call
1026 14:59:51.549152 (+ 75890us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:51.574288 22700 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:51.580248 22725 rpcz_store.cc:276] 1026 14:59:48.376363 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:48.380142 (+ 3779us) service_pool.cc:225] Handling call
1026 14:59:48.380442 (+ 300us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:48.543651 (+163209us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:51.570770 (+3027119us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b540040d3d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:51.570976 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":324300672}
I20251026 14:59:51.481563 25546 consensus_queue.cc:1059] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20251026 14:59:51.587734 22801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1379) took 1240 ms. Trace:
I20251026 14:59:51.589073 22801 rpcz_store.cc:276] 1026 14:59:50.347354 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.547026 (+1199672us) service_pool.cc:225] Handling call
1026 14:59:51.587669 (+ 40643us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:51.590234 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:51.590358 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:59:51.590643 22651 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.855s user 0.050s sys 0.054s
I20251026 14:59:51.590919 22801 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:51.587729 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1667) took 1205 ms. Trace:
W20251026 14:59:51.591048 22651 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.856s user 0.050s sys 0.054s
I20251026 14:59:51.591569 22734 rpcz_store.cc:276] 1026 14:59:50.382189 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.567199 (+1185010us) service_pool.cc:225] Handling call
1026 14:59:51.567520 (+ 321us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:51.587371 (+ 19851us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:51.587656 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:51.591046 22808 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:51.469419 25638 raft_consensus.cc:515] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:51.546722 25481 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:51.596225 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:51.596696 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:51.540459 25711 raft_consensus.cc:697] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [term 9 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:51.608476 22655 raft_consensus.cc:2468] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
I20251026 14:59:51.609792 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:51.610034 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:51.610558 22801 raft_consensus.cc:2468] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:51.611037 22655 raft_consensus.cc:2468] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:51.611635 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:51.611956 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 11 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:51.612619 22655 raft_consensus.cc:2468] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 10.
I20251026 14:59:51.612769 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:51.613636 22801 raft_consensus.cc:2393] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 9: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:51.614041 22801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1376) took 1267 ms. Trace:
W20251026 14:59:51.615507 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:51.627229 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:51.561403 25601 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:51.577456 25517 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:51.646281 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 120946us
I20251026 14:59:51.540045 25613 consensus_queue.cc:1048] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20251026 14:59:51.595489 25095 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:51.579444 25586 raft_consensus.cc:740] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:51.656134 25517 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:51.657043 25601 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:51.644567 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:51.603066 25717 raft_consensus.cc:3055] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 4 LEADER]: Stepping down as leader of term 4
W20251026 14:59:51.667655 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:51.667559 22808 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
I20251026 14:59:51.576496 25713 consensus_queue.cc:1059] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.102s
I20251026 14:59:51.579118 25583 consensus_queue.cc:1048] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
I20251026 14:59:51.692914 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:51.695809 25583 raft_consensus.cc:2804] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:51.695871 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:51.696389 25583 raft_consensus.cc:493] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:51.696622 22655 raft_consensus.cc:2468] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
I20251026 14:59:51.696736 22806 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:51.696828 25583 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:51.697669 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:51.698191 25401 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:51.698843 25401 raft_consensus.cc:2804] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251026 14:59:51.699213 25401 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:51.699579 25401 raft_consensus.cc:3060] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:51.699664 25196 raft_consensus.cc:3055] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 10 LEADER]: Stepping down as leader of term 10
I20251026 14:59:51.700222 25196 raft_consensus.cc:740] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:51.702605 22729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1602) took 5369 ms. Trace:
I20251026 14:59:51.703341 25723 raft_consensus.cc:2804] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader election won for term 9
I20251026 14:59:51.703176 22729 rpcz_store.cc:276] 1026 14:59:46.337248 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:46.340831 (+ 3583us) service_pool.cc:225] Handling call
1026 14:59:46.341142 (+ 311us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:48.845940 (+2504798us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:51.702197 (+2856257us) spinlock_profiling.cc:243] Waited 2.42 s on lock 0x7b54000b9ad4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:51.702485 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5574652672}
I20251026 14:59:51.704185 25723 raft_consensus.cc:697] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [term 9 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:51.693778 25517 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:51.711714 22658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1587) took 4749 ms. Trace:
I20251026 14:59:51.712177 22658 rpcz_store.cc:276] 1026 14:59:46.962151 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:46.962280 (+ 129us) service_pool.cc:225] Handling call
1026 14:59:49.454033 (+2491753us) spinlock_profiling.cc:243] Waited 7.44 ms on lock 0x7b5800000998. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8ce074451 00007fd8ce074302 00007fd8d6f0dad9 00007fd8d6090d30 00007fd8d615984b 00007fd8d61598d1 00007fd8d611b580 00007fd8d6105ee9 00007fd8d6105992 00007fd8cf69fd34
1026 14:59:51.479218 (+2025185us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:51.540913 (+ 61695us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:51.711445 (+170532us) spinlock_profiling.cc:243] Waited 2.02 s on lock 0x7b54001d92d0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:51.711613 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4673903232}
W20251026 14:59:51.713762 23400 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43606 (request call id 84) took 2358 ms (client timeout 1731 ms). Trace:
W20251026 14:59:51.714246 23400 rpcz_store.cc:269] 1026 14:59:49.355666 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:49.369369 (+ 13703us) service_pool.cc:225] Handling call
1026 14:59:51.713550 (+2344181us) spinlock_profiling.cc:243] Waited 9.58 ms on lock 0x7b54005aebd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:51.713680 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22035328}
I20251026 14:59:51.715624 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:51.715965 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32888 (request call id 152) took 2120 ms. Trace:
I20251026 14:59:51.716517 25638 leader_election.cc:290] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:51.717528 22802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1370) took 1451 ms. Trace:
I20251026 14:59:51.717890 22802 rpcz_store.cc:276] 1026 14:59:50.275023 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.644212 (+1369189us) service_pool.cc:225] Handling call
1026 14:59:51.717448 (+ 73236us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:51.675174 25545 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:51.640035 25647 consensus_queue.cc:1059] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.079s
I20251026 14:59:51.716413 22659 rpcz_store.cc:276] 1026 14:59:49.676725 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:50.660209 (+983484us) service_pool.cc:225] Handling call
1026 14:59:50.661052 (+ 843us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:50.674395 (+ 13343us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:50.674431 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:50.674474 (+ 43us) raft_consensus.cc:1645] Marking committed up to 7
1026 14:59:50.674521 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:51.715722 (+1041201us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b540012e3d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:51.715769 (+ 47us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:51.715903 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":29808384}
I20251026 14:59:51.721338 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:51.723953 22807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1359) took 1656 ms. Trace:
I20251026 14:59:51.705389 25723 consensus_queue.cc:237] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:51.675196 25717 raft_consensus.cc:740] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
W20251026 14:59:51.732511 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 86875.9us
I20251026 14:59:51.734838 22659 raft_consensus.cc:2410] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251026 14:59:51.706393 25570 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:51.707463 25615 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:51.614346 22801 rpcz_store.cc:276] 1026 14:59:50.346310 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.612549 (+1266239us) service_pool.cc:225] Handling call
1026 14:59:51.613985 (+ 1436us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:51.736114 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:51.737802 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 236871us
I20251026 14:59:51.643867 25506 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:51.724493 22807 rpcz_store.cc:276] 1026 14:59:50.067266 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:50.067400 (+ 134us) service_pool.cc:225] Handling call
1026 14:59:50.266107 (+198707us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:50.430954 (+164847us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:50.430992 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:50.431529 (+ 537us) log.cc:844] Serialized 26 byte log entry
1026 14:59:51.325197 (+893668us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:51.534563 (+209366us) log.cc:844] Serialized 14 byte log entry
1026 14:59:51.576363 (+ 41800us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:51.576473 (+ 110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:51.723641 (+147168us) raft_consensus.cc:1685] finished
1026 14:59:51.723672 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:51.723877 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":72555,"raft.queue_time_us":494096,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":20471,"thread_start_us":1151149,"threads_started":3,"wal-append.queue_time_us":1258453}
I20251026 14:59:51.742833 22802 raft_consensus.cc:1240] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20251026 14:59:51.744104 22802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1362) took 1668 ms. Trace:
I20251026 14:59:51.744529 22802 rpcz_store.cc:276] 1026 14:59:50.075302 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.742003 (+1666701us) service_pool.cc:225] Handling call
1026 14:59:51.742507 (+ 504us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:51.743793 (+ 1286us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:51.744036 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:51.751992 22771 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.125s user 0.012s sys 0.056s
W20251026 14:59:51.752643 22771 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.155s user 0.013s sys 0.056s
I20251026 14:59:51.702684 25586 consensus_queue.cc:260] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:51.761343 25713 consensus_queue.cc:1243] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer 562bb1e1c8e445b8baecd29c3669658e log is divergent from this leader: its last log entry 11.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20251026 14:59:51.769439 25519 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:51.774801 25647 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:51.776568 22806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1381) took 1322 ms. Trace:
I20251026 14:59:51.777714 25711 consensus_queue.cc:237] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:51.752697 25570 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:51.778914 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:51.818476 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:51.819032 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:51.819550 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:51.779385 25725 log.cc:927] Time spent T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.121s user 0.002s sys 0.000s
W20251026 14:59:51.820026 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:51.776968 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 43509.9us
I20251026 14:59:51.782361 25604 raft_consensus.cc:3055] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:51.762041 25546 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:51.787907 22726 raft_consensus.cc:1240] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 9. Current term is 10. Ops: [9.4-9.4]
I20251026 14:59:51.787322 22728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1656) took 2026 ms. Trace:
I20251026 14:59:51.787426 25583 raft_consensus.cc:515] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:51.793229 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1658) took 2031 ms. Trace:
W20251026 14:59:51.798712 22727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35092 (request call id 1644) took 3420 ms (client timeout 3198 ms). Trace:
I20251026 14:59:51.802136 25601 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:51.805243 22802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1365) took 1729 ms. Trace:
I20251026 14:59:51.813654 22806 rpcz_store.cc:276] 1026 14:59:50.453997 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:50.852585 (+398588us) service_pool.cc:225] Handling call
1026 14:59:51.776391 (+923806us) spinlock_profiling.cc:243] Waited 843 ms on lock 0x7b54001bd354. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:51.776505 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1939603200}
W20251026 14:59:51.820433 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:51.837157 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:51.822229 22622 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:51.827178 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
W20251026 14:59:51.831754 22727 rpcz_store.cc:269] 1026 14:59:48.378520 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:48.419465 (+ 40945us) service_pool.cc:225] Handling call
1026 14:59:51.798470 (+3379005us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b54002dd054. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:51.798624 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":359317888}
I20251026 14:59:51.831645 22732 rpcz_store.cc:276] 1026 14:59:49.761742 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:50.645489 (+883747us) service_pool.cc:225] Handling call
1026 14:59:50.658495 (+ 13006us) raft_consensus.cc:1471] Updating replica for 4 ops
1026 14:59:50.684969 (+ 26474us) raft_consensus.cc:1534] Early marking committed up to index 2
1026 14:59:50.685001 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 3 ops
1026 14:59:50.685749 (+ 748us) log.cc:844] Serialized 78 byte log entry
1026 14:59:50.973534 (+287785us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:50.973593 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:50.973680 (+ 87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:51.792935 (+819255us) raft_consensus.cc:1685] finished
1026 14:59:51.792961 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:51.793157 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12159,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":95,"thread_start_us":282838,"threads_started":1,"wal-append.queue_time_us":1092809}
I20251026 14:59:51.835137 23404 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:51.829625 22728 rpcz_store.cc:276] 1026 14:59:49.761172 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:50.590939 (+829767us) service_pool.cc:225] Handling call
1026 14:59:50.591311 (+ 372us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:50.591741 (+ 430us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:50.591784 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:50.630329 (+ 38545us) log.cc:844] Serialized 26 byte log entry
1026 14:59:51.346806 (+716477us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:51.346941 (+ 135us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:51.347017 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:51.787030 (+440013us) raft_consensus.cc:1685] finished
1026 14:59:51.787056 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:51.787249 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11084,"raft.queue_time_us":568,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":9479,"thread_start_us":447980,"threads_started":1,"wal-append.queue_time_us":1135808}
I20251026 14:59:51.675820 25095 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:51.836515 22802 rpcz_store.cc:276] 1026 14:59:50.076189 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.745535 (+1669346us) service_pool.cc:225] Handling call
1026 14:59:51.804556 (+ 59021us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:51.804816 (+ 260us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:51.804853 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:51.804911 (+ 58us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:51.804957 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:51.805030 (+ 73us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:51.805167 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:51.845827 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 466885us
I20251026 14:59:51.828526 25519 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:51.894490 25519 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:51.894958 23404 raft_consensus.cc:1240] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:51.896221 23404 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:51.896713 23401 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:51.897004 25519 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:51.831662 25405 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:51.878955 25193 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:51.884100 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:51.868816 25601 raft_consensus.cc:2804] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:51.887987 25095 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:51.888293 22653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:54560 (request call id 259) took 2415 ms. Trace:
I20251026 14:59:51.935475 22653 rpcz_store.cc:276] 1026 14:59:49.492469 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:50.707853 (+1215384us) service_pool.cc:225] Handling call
1026 14:59:51.887648 (+1179795us) spinlock_profiling.cc:243] Waited 39.4 ms on lock 0x7b54000dbe54. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c7fe0 00007fd8d1166aca 00007fd8d610b01e 00007fd8cf6b7437 00007fd8cf6b73c2 00007fd8cf6b72f1 00007fd8cf6b726c 00007fd8cf6b6522 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:51.888167 (+ 519us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":90514688}
I20251026 14:59:51.891273 25570 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:51.891669 22800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1357) took 1873 ms. Trace:
I20251026 14:59:51.893177 25583 leader_election.cc:290] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:51.823258 25604 raft_consensus.cc:740] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:51.893703 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1398) took 3046 ms. Trace:
I20251026 14:59:51.828023 25611 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:51.871860 22697 leader_election.cc:400] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 6. Current term is 7.
I20251026 14:59:51.953469 22800 rpcz_store.cc:276] 1026 14:59:50.018442 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:50.018572 (+ 130us) service_pool.cc:225] Handling call
1026 14:59:50.018869 (+ 297us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:50.439632 (+420763us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:50.439680 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:50.449255 (+ 9575us) log.cc:844] Serialized 26 byte log entry
1026 14:59:51.470649 (+1021394us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:51.470694 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:51.470783 (+ 89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:51.778785 (+308002us) spinlock_profiling.cc:243] Waited 6.67 ms on lock 0x7b440002304c. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8cdfc0905 00007fd8cdfc1322 00007fd8d0d0a231 00007fd8d0d05b5a 00007fd8d0cf991e 00007fd8d0cf8c66 00007fd8d0cf8888 00007fd8d0d022c1 00007fd8d0d02279 00007fd8d0d02209
1026 14:59:51.891233 (+112448us) raft_consensus.cc:1685] finished
1026 14:59:51.891260 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:51.891584 (+ 324us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":112,"raft.run_cpu_time_us":61,"raft.run_wall_time_us":60,"spinlock_wait_cycles":15343488,"thread_start_us":1324114,"threads_started":2,"wal-append.queue_time_us":1033978}
I20251026 14:59:51.974231 22697 leader_election.cc:403] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
W20251026 14:59:51.975087 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 103433us
I20251026 14:59:51.975550 22800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49652 (request call id 361) took 1100 ms. Trace:
W20251026 14:59:51.838583 25646 consensus_meta.cc:287] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 1.034s user 0.010s sys 0.003s
I20251026 14:59:51.818034 25647 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:51.897488 23401 raft_consensus.cc:2468] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:51.908388 25539 raft_consensus.cc:493] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:51.908915 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:51.909404 25390 raft_consensus.cc:2749] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20251026 14:59:51.893028 25621 raft_consensus.cc:3055] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:51.831193 25635 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:51.909070 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:51.835029 25546 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:51.917995 25193 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:51.918308 25667 raft_consensus.cc:3060] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:51.959769 22651 rpcz_store.cc:276] 1026 14:59:48.847275 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:49.155261 (+307986us) service_pool.cc:225] Handling call
1026 14:59:49.155741 (+ 480us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:49.257465 (+101724us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:49.257495 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:49.257954 (+ 459us) log.cc:844] Serialized 26 byte log entry
1026 14:59:50.735274 (+1477320us) spinlock_profiling.cc:243] Waited 1.15 ms on lock 0x7b1800361a20. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce0942f7 00007fd8d6e6d219 00007fd8ce0b0c4b 00007fd8ce0aeeef 00007fd8ce0b11b2 00007fd8d0cf82bb 00007fd8d0cfddc3 00007fd8d0cfe171 00007fd8d111965f
1026 14:59:51.591873 (+856599us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:51.591914 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:51.591982 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:51.893384 (+301402us) raft_consensus.cc:1685] finished
1026 14:59:51.893407 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:51.893640 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":101292,"raft.queue_time_us":101664,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":7729,"spinlock_wait_cycles":2636672,"thread_start_us":856385,"threads_started":1,"wal-append.queue_time_us":2613303}
W20251026 14:59:51.960100 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 385936us
W20251026 14:59:52.020766 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 283667us
I20251026 14:59:51.975909 22800 rpcz_store.cc:276] 1026 14:59:50.877432 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.974706 (+1097274us) service_pool.cc:225] Handling call
1026 14:59:51.975021 (+ 315us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:51.975215 (+ 194us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:51.975255 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:51.975295 (+ 40us) raft_consensus.cc:1645] Marking committed up to 7
1026 14:59:51.975332 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:51.975385 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:51.975499 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:52.024169 22800 raft_consensus.cc:1240] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 3. Current term is 4. Ops: []
I20251026 14:59:51.923595 25601 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:52.002939 23404 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:52.003161 22652 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:52.008352 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:51.918735 25405 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:51.984836 25241 raft_consensus.cc:2749] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for earlier term 6. Current term is 7.
I20251026 14:59:51.838191 25717 consensus_queue.cc:260] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:52.018733 22771 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters:
W20251026 14:59:52.022081 22604 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 524d8339d5244ca98e86f0ff2fd02b67, TS 0d684445287b450a8f738b6ad4b59821 -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:52.031278 22604 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 524d8339d5244ca98e86f0ff2fd02b67, TS 0d684445287b450a8f738b6ad4b59821 -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:52.021747 22700 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:52.025024 22800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1481) took 1147 ms. Trace:
I20251026 14:59:52.031963 22800 rpcz_store.cc:276] 1026 14:59:50.877932 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:52.023516 (+1145584us) service_pool.cc:225] Handling call
1026 14:59:52.023892 (+ 376us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:52.024704 (+ 812us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:52.024952 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:51.987504 25287 raft_consensus.cc:697] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:52.010883 25621 raft_consensus.cc:740] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:52.023291 25604 consensus_queue.cc:260] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:52.034560 25522 consensus_queue.cc:1048] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:59:52.035336 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:51.987479 25611 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:52.036190 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:52.037340 22774 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:52.048878 22651 raft_consensus.cc:2468] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:52.050227 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:52.051123 22651 raft_consensus.cc:2393] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 7: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:52.007661 25539 raft_consensus.cc:515] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:52.043226 23369 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:52.045663 22653 raft_consensus.cc:2393] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 8: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:52.035794 22700 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:52.046310 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:52.034026 22697 connection.cc:232] RPC call timeout handler was delayed by -2.1469284429999789s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15810000000000002s for a response
I20251026 14:59:52.007534 25534 consensus_queue.cc:1059] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.026s
I20251026 14:59:52.029157 25601 raft_consensus.cc:3060] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:52.027585 25124 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:52.000458 25570 consensus_queue.cc:1059] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.025s
W20251026 14:59:52.062090 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 229579us
I20251026 14:59:52.063930 22774 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:52.018186 25635 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:52.044921 25735 raft_consensus.cc:2804] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election won for term 9
W20251026 14:59:52.048439 25704 consensus_meta.cc:287] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.522s user 0.006s sys 0.006s
I20251026 14:59:52.057404 25604 raft_consensus.cc:3060] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:52.067591 22700 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:52.027238 24740 consensus_queue.cc:1048] T 3620632bd772435f97f799be96d493b8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.031s
I20251026 14:59:52.026010 25369 consensus_queue.cc:1048] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20251026 14:59:52.024960 25707 consensus_queue.cc:1048] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20251026 14:59:52.046353 25545 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:52.046624 25713 raft_consensus.cc:2804] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20251026 14:59:52.072268 22697 connection.cc:232] RPC call timeout handler was delayed by -2.0493614499999921s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1807s for a response
I20251026 14:59:52.089956 25713 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:52.090154 22697 connection.cc:232] RPC call timeout handler was delayed by -1.9945847419999154s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1831s for a response
I20251026 14:59:52.090428 25713 raft_consensus.cc:3060] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
W20251026 14:59:52.090530 22697 connection.cc:232] RPC call timeout handler was delayed by -1.6788634749999574s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2048s for a response
W20251026 14:59:52.090925 22697 connection.cc:232] RPC call timeout handler was delayed by -1.5204634749999286s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22240000000000004s for a response
W20251026 14:59:52.091230 22697 connection.cc:232] RPC call timeout handler was delayed by -1.4587308620000385s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2202s for a response
I20251026 14:59:52.091869 22697 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:52.093361 25737 consensus_queue.cc:1048] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.008s
I20251026 14:59:52.094055 25536 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:52.073195 25241 raft_consensus.cc:2804] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:52.078599 22700 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:52.080106 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:52.084632 22659 raft_consensus.cc:2410] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20251026 14:59:52.084491 22774 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:52.044396 25540 consensus_queue.cc:1048] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20251026 14:59:52.068717 25733 raft_consensus.cc:2749] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251026 14:59:52.077634 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:52.102391 22651 raft_consensus.cc:2410] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20251026 14:59:52.150239 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 48529.2us
W20251026 14:59:52.151309 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:52.079088 25735 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:52.086627 25723 raft_consensus.cc:2749] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251026 14:59:52.164295 22774 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:52.097106 25573 raft_consensus.cc:3060] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:52.149923 25598 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:52.168571 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:52.086941 25732 raft_consensus.cc:2804] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20251026 14:59:52.173915 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:52.151038 25241 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:59:52.183796 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:52.083353 25124 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:52.160980 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 82413.7us
I20251026 14:59:52.184592 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:52.177814 22774 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:52.096284 25287 consensus_queue.cc:237] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:52.084506 25621 consensus_queue.cc:260] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:52.160072 25534 raft_consensus.cc:2804] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:52.184053 25390 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:52.189441 25733 raft_consensus.cc:2804] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:52.184085 25732 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:59:52.195538 22697 leader_election.cc:336] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Timed out: RequestConsensusVote RPC to 127.21.2.3:39645 timed out after 2.048s (SENT)
I20251026 14:59:52.099284 25545 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:52.187539 25241 raft_consensus.cc:3060] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:52.157573 25735 raft_consensus.cc:3060] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:52.192389 23369 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:52.172233 25186 consensus_queue.cc:1048] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20251026 14:59:52.183388 25570 consensus_queue.cc:1048] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.026s
W20251026 14:59:52.217726 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:52.218505 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:52.219051 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:52.210553 22697 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:52.197219 25704 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:52.214025 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:52.204838 25707 raft_consensus.cc:2749] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251026 14:59:52.210247 25481 consensus_meta.cc:287] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.567s user 0.012s sys 0.000s
I20251026 14:59:52.214800 22731 raft_consensus.cc:1275] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251026 14:59:52.211889 25405 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:52.215565 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:52.184548 25715 consensus_queue.cc:1048] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20251026 14:59:52.221107 22651 raft_consensus.cc:1275] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251026 14:59:52.213888 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:52.204730 25534 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:52.247882 22800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1482) took 1369 ms. Trace:
I20251026 14:59:52.240262 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:52.249904 22804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1360) took 2175 ms. Trace:
I20251026 14:59:52.188933 25401 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:52.247618 24740 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:52.247996 22730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 1243) took 2377 ms (client timeout 1737 ms). Trace:
W20251026 14:59:52.304337 22730 rpcz_store.cc:269] 1026 14:59:49.871304 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:50.304104 (+432800us) service_pool.cc:225] Handling call
1026 14:59:52.247778 (+1943674us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b54000b90d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:52.247920 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":319265664}
I20251026 14:59:52.246512 25592 raft_consensus.cc:493] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:52.239806 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:52.239123 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:52.210208 25733 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:52.287456 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:52.323341 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 12 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:52.248967 22657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:46598 (request call id 13) took 5648 ms. Trace:
I20251026 14:59:52.203744 25753 raft_consensus.cc:2804] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:52.212379 25546 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:52.297183 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:52.213110 25647 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:52.215682 25307 raft_consensus.cc:2749] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:52.298141 22800 rpcz_store.cc:276] 1026 14:59:50.878259 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:52.032822 (+1154563us) service_pool.cc:225] Handling call
1026 14:59:52.033143 (+ 321us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:52.041066 (+ 7923us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:52.041096 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:52.047155 (+ 6059us) log.cc:844] Serialized 26 byte log entry
1026 14:59:52.182766 (+135611us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:52.182809 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:52.182890 (+ 81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:52.247659 (+ 64769us) raft_consensus.cc:1685] finished
1026 14:59:52.247681 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:52.247821 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21025,"raft.queue_time_us":7836,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":8067,"thread_start_us":103039,"threads_started":1,"wal-append.queue_time_us":179863}
I20251026 14:59:52.371718 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:52.372560 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:52.373225 22730 raft_consensus.cc:2376] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 9. Re-sending same reply.
I20251026 14:59:52.373370 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 12 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:52.301692 22804 rpcz_store.cc:276] 1026 14:59:50.074690 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.749692 (+1675002us) service_pool.cc:225] Handling call
1026 14:59:51.765882 (+ 16190us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:51.810427 (+ 44545us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:51.810464 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:51.825988 (+ 15524us) log.cc:844] Serialized 26 byte log entry
1026 14:59:52.073559 (+247571us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:52.073601 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:52.073677 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:52.249678 (+176001us) raft_consensus.cc:1685] finished
1026 14:59:52.249702 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:52.249840 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43749,"raft.queue_time_us":45086,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":120,"thread_start_us":179595,"threads_started":1,"wal-append.queue_time_us":317507}
W20251026 14:59:52.302748 22697 leader_election.cc:336] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Timed out: RequestConsensusVote RPC to 127.21.2.3:39645 timed out after 2.224s (SENT)
W20251026 14:59:52.325261 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:52.326323 22657 rpcz_store.cc:276] 1026 14:59:46.600856 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:46.601039 (+ 183us) service_pool.cc:225] Handling call
1026 14:59:52.248712 (+5647673us) spinlock_profiling.cc:243] Waited 2.89 s on lock 0x7b540034c454. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d11583f3 00007fd8d6109bca 00007fd8cf6b2934 00007fd8cf6b28c2 00007fd8cf6b27f1 00007fd8cf6b276c 00007fd8cf6b1a22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:52.248859 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6635690368}
I20251026 14:59:52.375097 22697 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:52.376075 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:52.376760 22657 raft_consensus.cc:2376] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7. Re-sending same reply.
I20251026 14:59:52.301635 25534 raft_consensus.cc:3060] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
W20251026 14:59:52.350575 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:52.300002 25743 log.cc:927] Time spent T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.134s user 0.001s sys 0.001s
I20251026 14:59:52.223495 25752 raft_consensus.cc:2749] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:52.210909 25732 raft_consensus.cc:3060] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Advancing to term 11
I20251026 14:59:52.210619 25390 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:52.222218 25570 consensus_queue.cc:1243] T 524d8339d5244ca98e86f0ff2fd02b67 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer df3a0d1619ee48e8ab68ee0543a72a29 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20251026 14:59:52.366385 25611 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:52.238643 25522 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
W20251026 14:59:52.358526 25739 log.cc:927] Time spent T 00000000000000000000000000000000 P a9e76e22adc1466cac02911e2229dc0d: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
I20251026 14:59:52.358947 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:52.415457 22725 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:52.359541 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:52.360385 25401 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:52.322073 24740 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:52.361994 25747 log.cc:927] Time spent T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1: Append to log took a long time: real 0.163s user 0.000s sys 0.000s
I20251026 14:59:52.363412 25713 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:52.286701 25481 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:52.380687 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:52.382136 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:52.429121 22729 raft_consensus.cc:2393] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 9: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:52.324222 22651 raft_consensus.cc:2468] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 11.
I20251026 14:59:52.217973 25214 consensus_queue.cc:1048] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.148s
I20251026 14:59:52.432694 22659 raft_consensus.cc:2410] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20251026 14:59:52.392143 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:52.444751 22733 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
W20251026 14:59:52.393335 25742 log.cc:927] Time spent T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.209s user 0.000s sys 0.001s
I20251026 14:59:52.394012 22807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1364) took 2318 ms. Trace:
I20251026 14:59:52.527031 22807 rpcz_store.cc:276] 1026 14:59:50.075912 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.743367 (+1667455us) service_pool.cc:225] Handling call
1026 14:59:51.743702 (+ 335us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:51.743933 (+ 231us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:51.743968 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:51.744645 (+ 677us) log.cc:844] Serialized 26 byte log entry
1026 14:59:52.011364 (+266719us) raft_consensus.cc:1645] Marking committed up to 6
1026 14:59:52.011407 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:52.011484 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:52.393781 (+382297us) raft_consensus.cc:1685] finished
1026 14:59:52.393802 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:52.393954 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":187417,"threads_started":1,"wal-append.queue_time_us":269698}
I20251026 14:59:52.395692 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:52.529964 22804 raft_consensus.cc:2376] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7. Re-sending same reply.
I20251026 14:59:52.251338 25704 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:52.326021 25733 raft_consensus.cc:3060] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:52.400265 22730 raft_consensus.cc:1275] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20251026 14:59:52.400799 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:52.402101 25755 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:52.365662 25753 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:52.286875 25638 consensus_queue.cc:1048] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20251026 14:59:52.401161 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:52.564307 22800 raft_consensus.cc:2468] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 2.
I20251026 14:59:52.565379 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:52.566628 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 11 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:52.567291 22774 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:52.366252 25539 leader_election.cc:290] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:52.390049 23368 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:52.366873 25707 raft_consensus.cc:493] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
W20251026 14:59:52.411645 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:52.394865 25635 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:52.326333 25592 raft_consensus.cc:515] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:52.454393 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:52.469322 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:52.499719 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:52.549754 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:52.546514 23369 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:52.551460 22657 raft_consensus.cc:2393] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 9: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:52.546514 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:52.601513 22659 raft_consensus.cc:2468] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 4.
I20251026 14:59:52.603214 22651 raft_consensus.cc:2468] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:52.419430 25522 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:52.542448 22729 raft_consensus.cc:1275] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:52.614141 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 1418) took 1150 ms. Trace:
I20251026 14:59:52.614717 22652 rpcz_store.cc:276] 1026 14:59:51.463740 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.468440 (+ 4700us) service_pool.cc:225] Handling call
1026 14:59:52.613963 (+1145523us) spinlock_profiling.cc:243] Waited 532 ms on lock 0x7b54000dc854. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:52.614076 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1222818048}
I20251026 14:59:52.616307 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:52.618414 22652 raft_consensus.cc:2410] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 10 because replica has last-logged OpId of term: 9 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 3.
I20251026 14:59:52.563104 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:52.431931 25672 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:52.567926 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:52.535920 25580 raft_consensus.cc:493] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:52.597509 22730 raft_consensus.cc:1240] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:52.431641 25530 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:52.604195 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:52.529098 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:52.629205 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:52.569432 25753 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:52.672541 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:52.587857 25540 raft_consensus.cc:2749] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20251026 14:59:52.609238 25750 log.cc:927] Time spent T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.398s user 0.000s sys 0.001s
I20251026 14:59:52.699210 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:52.692763 22800 raft_consensus.cc:2393] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 9: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:52.706966 22774 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:52.733078 22774 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:52.606984 25635 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:52.599634 25707 raft_consensus.cc:515] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:52.719548 22659 raft_consensus.cc:2376] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8. Re-sending same reply.
I20251026 14:59:52.726738 22657 raft_consensus.cc:3060] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:52.839608 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:52.840371 22659 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:52.844378 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:52.724746 25604 raft_consensus.cc:493] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:52.885735 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1239) took 3046 ms. Trace:
I20251026 14:59:52.885877 25604 raft_consensus.cc:515] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:52.886206 22726 rpcz_store.cc:276] 1026 14:59:49.839738 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:50.667188 (+827450us) service_pool.cc:225] Handling call
1026 14:59:51.787635 (+1120447us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:51.829205 (+ 41570us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:52.885514 (+1056309us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54001e8cd0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:52.885669 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2575755648}
I20251026 14:59:52.887964 25604 leader_election.cc:290] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:52.888767 22656 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32888 (request call id 153) took 3171 ms (client timeout 1768 ms). Trace:
I20251026 14:59:52.700642 25580 raft_consensus.cc:515] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:52.889205 22656 rpcz_store.cc:269] 1026 14:59:49.716966 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:50.263443 (+546477us) service_pool.cc:225] Handling call
1026 14:59:52.888562 (+2625119us) spinlock_profiling.cc:243] Waited 47.1 ms on lock 0x7b540010d1d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:52.888671 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":108304128}
I20251026 14:59:52.890389 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:52.891043 22656 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:52.891918 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:52.700137 25748 log.cc:927] Time spent T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.136s user 0.000s sys 0.001s
I20251026 14:59:52.736341 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:52.937290 22800 raft_consensus.cc:2376] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 10. Re-sending same reply.
I20251026 14:59:52.938337 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:52.744958 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:52.940709 22730 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:52.743966 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:52.693787 25545 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:52.888031 25617 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:52.943310 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:52.944011 22800 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:52.694845 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:52.748382 22804 raft_consensus.cc:2393] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 8: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:52.950071 22651 raft_consensus.cc:2410] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20251026 14:59:52.772428 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:52.951714 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:52.775561 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:52.829303 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:52.955624 25717 raft_consensus.cc:3060] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:52.956097 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:52.842621 25214 raft_consensus.cc:2804] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:52.884963 24740 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:52.842466 25570 raft_consensus.cc:2804] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20251026 14:59:52.896124 22650 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1590) took 5883 ms (client timeout 1631 ms). Trace:
I20251026 14:59:52.893045 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:52.979648 22656 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
W20251026 14:59:52.729240 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:52.979359 22650 rpcz_store.cc:269] 1026 14:59:46.963920 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:46.964076 (+ 156us) service_pool.cc:225] Handling call
1026 14:59:49.373347 (+2409271us) spinlock_profiling.cc:243] Waited 7.26 ms on lock 0x7b5800000998. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8ce074451 00007fd8ce074302 00007fd8d6f0dad9 00007fd8d6090d30 00007fd8d615984b 00007fd8d61598d1 00007fd8d6107660 00007fd8d6106f44 00007fd8cf6a4834 00007fd8cf6a47c2
1026 14:59:52.847053 (+3473706us) spinlock_profiling.cc:243] Waited 374 ms on lock 0x7b54002a94d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:52.847211 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":876691200}
I20251026 14:59:53.046365 22729 raft_consensus.cc:1240] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:53.051486 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:53.052456 22656 raft_consensus.cc:2410] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20251026 14:59:53.053104 22725 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.004s sys 0.004s
W20251026 14:59:53.060607 22725 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.538s user 0.004s sys 0.004s
I20251026 14:59:53.060686 22799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1462) took 3704 ms. Trace:
I20251026 14:59:53.061103 22799 rpcz_store.cc:276] 1026 14:59:49.356381 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:49.356510 (+ 129us) service_pool.cc:225] Handling call
1026 14:59:49.358262 (+ 1752us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:50.499102 (+1140840us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.060394 (+2561292us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b540020bcd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:53.060621 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2465112960}
I20251026 14:59:52.842240 25635 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:52.959662 22807 raft_consensus.cc:2410] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
W20251026 14:59:53.078868 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:53.083825 22650 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663"
dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
mode: GRACEFUL
new_leader_uuid: "562bb1e1c8e445b8baecd29c3669658e"
from {username='slave'} at 127.0.0.1:46598
W20251026 14:59:53.083503 22697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.707s user 0.024s sys 0.042s
I20251026 14:59:53.084563 22650 raft_consensus.cc:606] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Received request to transfer leadership to TS 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:53.085013 22697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.709s user 0.024s sys 0.042s
I20251026 14:59:53.085246 22650 raft_consensus.cc:614] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20251026 14:59:53.085635 22656 raft_consensus.cc:1275] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20251026 14:59:53.086372 22656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1401) took 1756 ms. Trace:
I20251026 14:59:53.086783 22656 rpcz_store.cc:276] 1026 14:59:51.329542 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.084967 (+1755425us) service_pool.cc:225] Handling call
1026 14:59:53.085314 (+ 347us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:53.086034 (+ 720us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.086300 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:53.090103 22652 raft_consensus.cc:1240] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20251026 14:59:53.092867 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1403) took 1762 ms. Trace:
I20251026 14:59:53.039849 25714 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:52.977679 25570 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:52.978159 25214 raft_consensus.cc:493] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:53.040771 25390 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:53.066476 25751 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.013s sys 0.015s
W20251026 14:59:53.130131 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:53.121755 22650 raft_consensus.cc:1240] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 9. Current term is 10. Ops: [9.6-9.6]
W20251026 14:59:53.138917 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 836145us
I20251026 14:59:53.147682 22650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1408) took 1797 ms. Trace:
I20251026 14:59:53.148344 22650 rpcz_store.cc:276] 1026 14:59:51.349871 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.113520 (+1763649us) service_pool.cc:225] Handling call
1026 14:59:53.121422 (+ 7902us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:53.147317 (+ 25895us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.147607 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:53.093521 22652 rpcz_store.cc:276] 1026 14:59:51.330009 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.089249 (+1759240us) service_pool.cc:225] Handling call
1026 14:59:53.089791 (+ 542us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:53.092384 (+ 2593us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.092789 (+ 405us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:52.951061 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:53.160714 22774 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:53.164482 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:53.166287 23369 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:53.139482 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:53.179387 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:53.137303 25751 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.708s user 0.014s sys 0.016s
I20251026 14:59:53.183442 22774 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:53.188270 22774 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:53.120996 25570 raft_consensus.cc:3060] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 4 FOLLOWER]: Advancing to term 5
W20251026 14:59:53.167907 23368 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:53.112694 25713 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:53.168715 25575 raft_consensus.cc:493] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:53.192109 21512 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251026 14:59:53.177860 25196 consensus_queue.cc:260] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:53.192920 25610 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:53.193063 25580 leader_election.cc:290] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:53.186282 25704 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:53.121884 25214 raft_consensus.cc:3060] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:53.193795 25481 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:53.130784 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:53.193840 25540 raft_consensus.cc:2749] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251026 14:59:53.208187 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:53.175578 25592 leader_election.cc:290] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:53.250630 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:53.230701 25580 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:53.184422 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:53.186545 25522 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:59:53.219542 22659 raft_consensus.cc:2468] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
I20251026 14:59:53.263013 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 1435) took 1056 ms. Trace:
I20251026 14:59:53.189375 25702 raft_consensus.cc:2804] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:53.263536 22659 rpcz_store.cc:276] 1026 14:59:52.206358 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:52.839278 (+632920us) service_pool.cc:225] Handling call
1026 14:59:53.262952 (+423674us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:53.264137 22804 raft_consensus.cc:2393] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 7: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:53.196695 22657 raft_consensus.cc:2468] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:53.264839 22657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1659) took 1037 ms. Trace:
I20251026 14:59:53.265362 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:53.265297 22657 rpcz_store.cc:276] 1026 14:59:52.227243 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:52.602839 (+375596us) service_pool.cc:225] Handling call
1026 14:59:53.264754 (+661915us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:53.267407 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1419) took 1744 ms. Trace:
I20251026 14:59:53.267580 22657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1417) took 1816 ms. Trace:
I20251026 14:59:53.267762 22659 rpcz_store.cc:276] 1026 14:59:51.522849 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.266671 (+1743822us) service_pool.cc:225] Handling call
1026 14:59:53.266932 (+ 261us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:53.267101 (+ 169us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:53.267132 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:53.267171 (+ 39us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:53.267202 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.267255 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:53.267356 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:53.122048 25714 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:53.212988 22774 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:53.214483 25708 raft_consensus.cc:2749] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:53.194978 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:53.273955 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1420) took 1725 ms. Trace:
I20251026 14:59:53.268289 22657 rpcz_store.cc:276] 1026 14:59:51.451732 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.266453 (+1814721us) service_pool.cc:225] Handling call
1026 14:59:53.266854 (+ 401us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:53.267130 (+ 276us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:53.267168 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:53.267229 (+ 61us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:53.267306 (+ 77us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.267383 (+ 77us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:53.267511 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:53.274534 22651 rpcz_store.cc:276] 1026 14:59:51.548794 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.273091 (+1724297us) service_pool.cc:225] Handling call
1026 14:59:53.273435 (+ 344us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:53.273610 (+ 175us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:53.273635 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:53.273685 (+ 50us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:53.273725 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.273784 (+ 59us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:53.273890 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:53.274718 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 67229.6us
I20251026 14:59:53.279392 22657 raft_consensus.cc:1240] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:53.280862 22657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1429) took 1729 ms. Trace:
I20251026 14:59:53.281967 22657 rpcz_store.cc:276] 1026 14:59:51.551460 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.277077 (+1725617us) service_pool.cc:225] Handling call
1026 14:59:53.279211 (+ 2134us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:53.280596 (+ 1385us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.280804 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:53.284089 22657 raft_consensus.cc:1240] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20251026 14:59:53.246249 22800 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 9.
I20251026 14:59:53.287295 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:53.252506 25124 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:53.262112 25196 raft_consensus.cc:3060] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Advancing to term 11
I20251026 14:59:53.217955 25575 raft_consensus.cc:515] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:53.275419 25536 consensus_meta.cc:287] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 1.181s user 0.008s sys 0.013s
I20251026 14:59:53.271584 25702 raft_consensus.cc:493] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:53.296067 25757 log.cc:927] Time spent T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.162s user 0.001s sys 0.001s
I20251026 14:59:53.287004 25707 leader_election.cc:290] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:53.271165 25580 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:53.319260 22731 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.748s user 0.029s sys 0.087s
W20251026 14:59:53.416282 22731 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.846s user 0.029s sys 0.107s
W20251026 14:59:53.400169 25193 consensus_meta.cc:287] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 1.375s user 0.022s sys 0.014s
W20251026 14:59:53.405076 25598 consensus_meta.cc:287] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 1.218s user 0.011s sys 0.015s
W20251026 14:59:53.344197 22734 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.939s user 0.050s sys 0.076s
W20251026 14:59:53.391295 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:53.443925 22734 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.038s user 0.051s sys 0.077s
W20251026 14:59:53.444146 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:53.277812 25704 raft_consensus.cc:493] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:53.394869 22657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32888 (request call id 156) took 1567 ms. Trace:
I20251026 14:59:53.394927 22800 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:53.408064 25702 raft_consensus.cc:3060] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
W20251026 14:59:53.395845 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:53.457787 25214 raft_consensus.cc:515] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:53.459748 25753 consensus_meta.cc:287] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.713s user 0.007s sys 0.004s
I20251026 14:59:53.460198 25214 leader_election.cc:290] T 2e6d661a639541b3a40bf3c1fc6c0a88 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:53.460186 25753 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:53.452534 25704 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:53.463181 25241 consensus_meta.cc:287] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 1.241s user 0.016s sys 0.002s
I20251026 14:59:53.463656 25241 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:53.464365 25704 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:53.464706 22727 raft_consensus.cc:1240] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:53.416913 23399 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 8 FOLLOWER]: Advancing to term 9
W20251026 14:59:53.382265 25546 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.828s user 0.016s sys 0.035s
I20251026 14:59:53.403992 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1684) took 1117 ms. Trace:
W20251026 14:59:53.399256 25573 consensus_meta.cc:287] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 1.210s user 0.022s sys 0.012s
I20251026 14:59:53.415778 22805 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
I20251026 14:59:53.287590 25522 raft_consensus.cc:2804] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:53.256819 23403 raft_consensus.cc:3060] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Advancing to term 7
W20251026 14:59:53.417928 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 248290us
I20251026 14:59:53.393090 25604 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:53.438863 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:53.394119 22774 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:53.451292 22657 rpcz_store.cc:276] 1026 14:59:51.827974 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.283504 (+1455530us) service_pool.cc:225] Handling call
1026 14:59:53.283873 (+ 369us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:53.295320 (+ 11447us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.394809 (+ 99489us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:53.465477 22732 raft_consensus.cc:1240] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 8. Current term is 9. Ops: []
I20251026 14:59:53.466447 22728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47996 (request call id 186) took 1371 ms. Trace:
W20251026 14:59:53.467305 22733 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.855s user 0.008s sys 0.035s
I20251026 14:59:53.395927 22651 raft_consensus.cc:1240] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 4. Current term is 5. Ops: []
W20251026 14:59:53.554008 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:53.611295 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32888 (request call id 154) took 1791 ms. Trace:
I20251026 14:59:53.538753 22731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1679) took 1293 ms. Trace:
W20251026 14:59:53.611969 22733 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.999s user 0.011s sys 0.038s
I20251026 14:59:53.450671 25598 raft_consensus.cc:2749] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251026 14:59:53.597504 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:53.613922 22655 raft_consensus.cc:2410] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20251026 14:59:53.481353 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:53.613229 22728 rpcz_store.cc:276] 1026 14:59:52.095405 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:52.095649 (+ 244us) service_pool.cc:225] Handling call
1026 14:59:52.096017 (+ 368us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:52.156119 (+ 60102us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:52.156151 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:52.156651 (+ 500us) log.cc:844] Serialized 26 byte log entry
1026 14:59:52.613963 (+457312us) raft_consensus.cc:1645] Marking committed up to 7
1026 14:59:52.614004 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:52.614068 (+ 64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:53.466148 (+852080us) raft_consensus.cc:1685] finished
1026 14:59:53.466184 (+ 36us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:53.466364 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1140,"raft.queue_time_us":61485,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":9109,"thread_start_us":504020,"threads_started":2,"wal-append.queue_time_us":965966}
W20251026 14:59:53.614625 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 360688us
I20251026 14:59:53.507586 22725 rpcz_store.cc:276] 1026 14:59:52.286176 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:52.416412 (+130236us) service_pool.cc:225] Handling call
1026 14:59:52.430579 (+ 14167us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:52.430838 (+ 259us) raft_consensus.cc:1534] Early marking committed up to index 5
1026 14:59:53.061216 (+630378us) log.cc:844] Serialized 14 byte log entry
1026 14:59:53.403443 (+342227us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:53.403561 (+ 118us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:53.403636 (+ 75us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.403738 (+ 102us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:53.403912 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":269236,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":1208,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":34649,"thread_start_us":864048,"threads_started":2}
I20251026 14:59:53.613193 22731 rpcz_store.cc:276] 1026 14:59:52.252571 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:52.337195 (+ 84624us) service_pool.cc:225] Handling call
1026 14:59:52.337675 (+ 480us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:52.338129 (+ 454us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:52.338165 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:52.367837 (+ 29672us) log.cc:844] Serialized 52 byte log entry
1026 14:59:53.416780 (+1048943us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:53.416849 (+ 69us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.416952 (+ 103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:53.538498 (+121546us) raft_consensus.cc:1685] finished
1026 14:59:53.538526 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:53.538685 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":63725,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"thread_start_us":846573,"threads_started":1,"wal-append.queue_time_us":967412}
W20251026 14:59:53.615154 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 221112us
I20251026 14:59:53.443367 25635 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:53.550235 25733 consensus_meta.cc:287] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.972s user 0.018s sys 0.002s
I20251026 14:59:53.622241 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:53.602856 25617 consensus_meta.cc:287] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 0.659s user 0.011s sys 0.008s
W20251026 14:59:53.610697 25534 consensus_meta.cc:287] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 1.206s user 0.018s sys 0.018s
I20251026 14:59:53.606416 25714 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:53.540356 25575 leader_election.cc:290] T 9423e278399742f48ca9ee203d7f92cb P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:53.442775 25193 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:53.612692 25761 log.cc:927] Time spent T 3ea63385804c40689502aefb2b695e78 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.211s user 0.001s sys 0.000s
W20251026 14:59:53.552587 25732 consensus_meta.cc:287] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 1.137s user 0.013s sys 0.004s
W20251026 14:59:53.502312 25735 consensus_meta.cc:287] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 1.287s user 0.011s sys 0.010s
W20251026 14:59:53.543272 25667 consensus_meta.cc:287] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 1.520s user 0.012s sys 0.007s
I20251026 14:59:53.624351 25636 raft_consensus.cc:493] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:53.613246 22651 rpcz_store.cc:276] 1026 14:59:51.819837 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.285939 (+1466102us) service_pool.cc:225] Handling call
1026 14:59:53.395652 (+109713us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:53.610995 (+215343us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.611222 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:53.706262 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5c9a78d6a2b442eabd005c8378b68733" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:53.611867 25522 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:53.636348 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:53.707607 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:53.715158 22728 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
W20251026 14:59:53.684526 25530 consensus_meta.cc:287] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 0.984s user 0.018s sys 0.013s
I20251026 14:59:53.603942 25758 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:53.637425 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:53.720589 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:53.638669 25214 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:53.633019 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:53.725687 22657 raft_consensus.cc:2393] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 10: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:53.544981 25241 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:53.620283 25604 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:53.466491 25570 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:53.688941 22733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1690) took 1301 ms. Trace:
I20251026 14:59:53.657085 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:53.507346 25389 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:53.637481 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:53.619776 25643 raft_consensus.cc:2804] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:53.709044 22654 raft_consensus.cc:2410] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 3.
I20251026 14:59:53.728224 22806 raft_consensus.cc:1275] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251026 14:59:53.685431 22803 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
I20251026 14:59:53.638031 25534 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:53.693747 25753 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:53.709713 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:53.651762 22697 leader_election.cc:304] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:53.634845 25523 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:53.689904 25601 consensus_meta.cc:287] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 1.606s user 0.014s sys 0.005s
W20251026 14:59:53.693416 23368 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:53.507280 25546 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.960s user 0.023s sys 0.039s
I20251026 14:59:53.717778 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
W20251026 14:59:53.689908 25717 consensus_meta.cc:287] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1: Time spent flushing consensus metadata: real 0.734s user 0.009s sys 0.003s
I20251026 14:59:53.726583 22651 raft_consensus.cc:1240] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: []
W20251026 14:59:53.560422 25755 consensus_meta.cc:287] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.983s user 0.012s sys 0.000s
I20251026 14:59:53.716218 25522 raft_consensus.cc:3060] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:53.728673 22725 raft_consensus.cc:2468] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:53.728312 22733 rpcz_store.cc:276] 1026 14:59:52.387907 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:52.445678 (+ 57771us) service_pool.cc:225] Handling call
1026 14:59:52.559161 (+113483us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:52.559705 (+ 544us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:52.559747 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:52.560509 (+ 762us) log.cc:844] Serialized 52 byte log entry
1026 14:59:53.613961 (+1053452us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:53.614005 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.614070 (+ 65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:53.688521 (+ 74451us) raft_consensus.cc:1685] finished
1026 14:59:53.688550 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:53.688816 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":175,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":17816,"thread_start_us":1001262,"threads_started":1,"wal-append.queue_time_us":1074931}
I20251026 14:59:53.637001 25733 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:53.631330 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:53.740113 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 88485.4us
I20251026 14:59:53.743522 22656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1404) took 2413 ms. Trace:
I20251026 14:59:53.760242 25733 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:53.745098 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1683) took 1491 ms. Trace:
I20251026 14:59:53.714395 25732 raft_consensus.cc:515] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:53.708292 25735 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:53.760989 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:53.760963 22734 rpcz_store.cc:276] 1026 14:59:52.253974 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:52.374410 (+120436us) service_pool.cc:225] Handling call
1026 14:59:52.374728 (+ 318us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:52.374971 (+ 243us) raft_consensus.cc:1534] Early marking committed up to index 4
1026 14:59:53.444536 (+1069565us) log.cc:844] Serialized 14 byte log entry
1026 14:59:53.740957 (+296421us) log.cc:844] Serialized 14 byte log entry
1026 14:59:53.744652 (+ 3695us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:53.744724 (+ 72us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:53.744792 (+ 68us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.744882 (+ 90us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:53.745029 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":318389,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":9922,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":94,"thread_start_us":1334508,"threads_started":2}
I20251026 14:59:53.742514 25559 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:53.751421 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:53.753006 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1646) took 1914 ms. Trace:
I20251026 14:59:53.765842 22651 rpcz_store.cc:276] 1026 14:59:51.838963 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.725914 (+1886951us) service_pool.cc:225] Handling call
1026 14:59:53.726363 (+ 449us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:53.752718 (+ 26355us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.752948 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:53.729655 25214 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:53.736268 25576 raft_consensus.cc:493] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:53.735868 25185 raft_consensus.cc:493] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:53.736577 25737 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:53.766919 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:53.767706 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:53.742885 25580 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:53.743993 25598 raft_consensus.cc:2749] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:53.724349 25758 raft_consensus.cc:515] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:53.810058 22800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1393) took 1517 ms. Trace:
I20251026 14:59:53.769999 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:53.772408 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:53.736049 25636 raft_consensus.cc:515] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:53.754196 25601 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:53.747810 25196 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:53.777678 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:53.753216 25643 raft_consensus.cc:697] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
W20251026 14:59:53.782569 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:53.788638 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:53.762817 25755 raft_consensus.cc:2749] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:53.779606 25576 raft_consensus.cc:515] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:53.810472 22800 rpcz_store.cc:276] 1026 14:59:52.293033 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.748595 (+1455562us) service_pool.cc:225] Handling call
1026 14:59:53.809979 (+ 61384us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:53.747016 25523 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:53.836910 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:53.838104 22725 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:53.837623 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:53.820187 22650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1415) took 2407 ms. Trace:
I20251026 14:59:53.838708 22800 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:53.814978 22651 raft_consensus.cc:1275] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20251026 14:59:53.809814 25241 consensus_queue.cc:1059] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
W20251026 14:59:53.836326 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 217364us
I20251026 14:59:53.739071 25389 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:53.840086 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1651) took 1999 ms. Trace:
I20251026 14:59:53.790086 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:53.838956 22650 rpcz_store.cc:276] 1026 14:59:51.412599 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.149589 (+1736990us) service_pool.cc:225] Handling call
1026 14:59:53.184126 (+ 34537us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:53.184490 (+ 364us) raft_consensus.cc:1534] Early marking committed up to index 2
1026 14:59:53.184525 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:53.192243 (+ 7718us) log.cc:844] Serialized 26 byte log entry
1026 14:59:53.481709 (+289466us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:53.481773 (+ 64us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.481858 (+ 85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:53.819943 (+338085us) raft_consensus.cc:1685] finished
1026 14:59:53.819966 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:53.820122 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":80,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":7183,"thread_start_us":289057,"threads_started":1,"wal-append.queue_time_us":548871}
I20251026 14:59:53.845420 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:53.847467 22650 raft_consensus.cc:3060] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:53.838960 22733 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:53.840759 22651 rpcz_store.cc:276] 1026 14:59:51.840324 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.814197 (+1973873us) service_pool.cc:225] Handling call
1026 14:59:53.814558 (+ 361us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:53.839355 (+ 24797us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.840033 (+ 678us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:53.771481 25559 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:53.853950 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:53.851023 22651 raft_consensus.cc:1275] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251026 14:59:53.788313 25185 raft_consensus.cc:515] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:53.856498 25534 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:53.862237 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:53.857422 22653 raft_consensus.cc:2468] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 6.
I20251026 14:59:53.861389 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1652) took 2020 ms. Trace:
W20251026 14:59:53.863071 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 71629.9us
I20251026 14:59:53.863106 22651 rpcz_store.cc:276] 1026 14:59:51.840674 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.850420 (+2009746us) service_pool.cc:225] Handling call
1026 14:59:53.850715 (+ 295us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:53.860445 (+ 9730us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.861309 (+ 864us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:53.864717 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:53.867476 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:53.851814 25196 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:53.868738 25601 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Requested vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:53.870850 25196 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:53.872009 25586 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:53.872958 25570 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:53.858191 25610 consensus_queue.cc:1048] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
W20251026 14:59:53.876363 22805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1324) took 7439 ms (client timeout 2224 ms). Trace:
I20251026 14:59:53.880609 22729 raft_consensus.cc:1275] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20251026 14:59:53.882151 25604 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:53.883280 25193 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:53.884335 25735 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:53.892247 25702 raft_consensus.cc:515] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:53.895756 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:53.880609 22802 raft_consensus.cc:1275] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 11 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20251026 14:59:53.901788 25534 consensus_queue.cc:1048] T 4dea3b45142b41baacfa105498c9e04e P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:59:53.903474 22697 leader_election.cc:304] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:53.920368 25775 log.cc:927] Time spent T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20251026 14:59:53.924465 25702 leader_election.cc:290] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:53.925060 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32888 (request call id 155) took 2098 ms. Trace:
I20251026 14:59:53.925717 22659 rpcz_store.cc:276] 1026 14:59:51.826436 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.282029 (+1455593us) service_pool.cc:225] Handling call
1026 14:59:53.282458 (+ 429us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:53.285568 (+ 3110us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:53.389997 (+104429us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:53.391269 (+ 1272us) log.cc:844] Serialized 26 byte log entry
1026 14:59:53.778879 (+387610us) raft_consensus.cc:1645] Marking committed up to 7
1026 14:59:53.778923 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.778981 (+ 58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:53.924746 (+145765us) raft_consensus.cc:1685] finished
1026 14:59:53.924778 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:53.924987 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2109,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":108,"thread_start_us":333368,"threads_started":1,"wal-append.queue_time_us":431543}
I20251026 14:59:53.927891 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:53.928529 22659 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:53.930512 22774 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:53.932052 22802 pending_rounds.cc:85] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821: Aborting all ops after (but not including) 5
I20251026 14:59:53.932443 22802 pending_rounds.cc:107] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20251026 14:59:53.932737 22802 raft_consensus.cc:2889] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251026 14:59:53.876976 22805 rpcz_store.cc:269] 1026 14:59:46.437188 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:46.756172 (+318984us) service_pool.cc:225] Handling call
1026 14:59:53.876054 (+7119882us) spinlock_profiling.cc:243] Waited 6.59 s on lock 0x7b54001bec54. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:53.876218 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15164463488}
W20251026 14:59:53.934008 25776 log.cc:927] Time spent T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20251026 14:59:53.934670 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:53.935905 25732 leader_election.cc:290] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 11 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:53.936244 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:53.867969 22734 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:53.940069 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:53.940541 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:53.941028 22734 raft_consensus.cc:2393] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 8: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:53.906625 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:53.907979 25543 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:53.909121 25214 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:53.911798 22801 raft_consensus.cc:1240] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:53.920169 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:53.955192 22697 leader_election.cc:304] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:53.955892 22728 raft_consensus.cc:2393] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 8: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:53.944166 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:53.957118 22733 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:53.937114 22805 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:53.955247 22801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1366) took 3878 ms. Trace:
I20251026 14:59:53.945386 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:53.957841 22697 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:53.958181 22801 rpcz_store.cc:276] 1026 14:59:50.076469 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.047050 (+2970581us) service_pool.cc:225] Handling call
1026 14:59:53.911339 (+864289us) spinlock_profiling.cc:243] Waited 94.2 ms on lock 0x7b5800002a98. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8ce074451 00007fd8ce074302 00007fd8d6f0dad9 00007fd8d6090d30 00007fd8d615984b 00007fd8d61598d1 00007fd8d611b580 00007fd8d6105ee9 00007fd8d6105992 00007fd8cf69fd34
1026 14:59:53.911592 (+ 253us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:53.954875 (+ 43283us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.955169 (+ 294us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":216706432}
I20251026 14:59:53.959533 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:53.960561 22801 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:53.962312 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:53.963315 22659 raft_consensus.cc:3060] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:53.966020 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:53.960601 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:53.967228 22651 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 9.
I20251026 14:59:53.968755 22725 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
I20251026 14:59:53.968657 22774 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:53.970315 22734 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:53.791947 25737 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:53.973119 22800 raft_consensus.cc:2410] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20251026 14:59:53.973439 22697 leader_election.cc:304] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:53.924527 23399 raft_consensus.cc:1275] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 9 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20251026 14:59:53.919464 25758 leader_election.cc:290] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:53.976766 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:53.974046 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:53.985489 22728 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:53.981732 22659 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:53.987015 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:53.987293 22659 raft_consensus.cc:1240] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20251026 14:59:53.988638 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1436) took 1779 ms. Trace:
I20251026 14:59:53.988835 22774 leader_election.cc:304] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:53.989042 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 1.79282e+06us
I20251026 14:59:53.989104 22659 rpcz_store.cc:276] 1026 14:59:52.209299 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.986802 (+1777503us) service_pool.cc:225] Handling call
1026 14:59:53.987063 (+ 261us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:53.988365 (+ 1302us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.988585 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:53.959533 25755 raft_consensus.cc:2804] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election won for term 7
W20251026 14:59:53.990268 23370 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:53.962322 25735 raft_consensus.cc:2804] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 14:59:53.991025 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:53.989180 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:53.976258 22697 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:53.994834 22659 raft_consensus.cc:1275] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20251026 14:59:53.965529 25196 raft_consensus.cc:2804] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20251026 14:59:54.005069 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:54.002919 22733 raft_consensus.cc:2410] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251026 14:59:54.005887 22801 raft_consensus.cc:2468] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
W20251026 14:59:54.007121 22803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44594 (request call id 1467) took 4648 ms (client timeout 3141 ms). Trace:
I20251026 14:59:54.007249 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.007864 22801 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
I20251026 14:59:54.007579 22697 leader_election.cc:304] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:54.003469 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1666) took 1766 ms. Trace:
I20251026 14:59:54.008895 22659 rpcz_store.cc:276] 1026 14:59:52.237550 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.990116 (+1752566us) service_pool.cc:225] Handling call
1026 14:59:53.990411 (+ 295us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:54.003238 (+ 12827us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:54.003423 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:53.981580 25732 raft_consensus.cc:2804] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Leader election won for term 10
I20251026 14:59:53.958424 25543 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:54.007531 22803 rpcz_store.cc:269] 1026 14:59:49.359064 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:49.362023 (+ 2959us) service_pool.cc:225] Handling call
1026 14:59:54.006971 (+4644948us) spinlock_profiling.cc:243] Waited 4.32 s on lock 0x7b54001b80d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:54.007061 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9940553984}
I20251026 14:59:54.021680 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:53.983491 25481 raft_consensus.cc:2749] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:54.023102 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.023753 22803 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:54.024820 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.000982 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:54.025632 22729 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:54.002775 22805 raft_consensus.cc:2468] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 5.
I20251026 14:59:54.017144 22650 raft_consensus.cc:2468] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:54.027570 22774 leader_election.cc:304] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:54.027766 22651 raft_consensus.cc:1240] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20251026 14:59:54.029520 22622 leader_election.cc:336] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645): Timed out: RequestConsensusVote RPC to 127.21.2.3:39645 timed out after 1.664s (SENT)
I20251026 14:59:53.977775 25214 raft_consensus.cc:2804] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251026 14:59:54.014675 22659 raft_consensus.cc:1275] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
W20251026 14:59:53.996723 25527 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20251026 14:59:53.999940 23400 raft_consensus.cc:2468] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5.
I20251026 14:59:54.004091 25735 raft_consensus.cc:493] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:54.018431 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:54.047892 22800 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:54.018016 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40914 (request call id 98) took 4280 ms. Trace:
I20251026 14:59:54.033850 25214 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:53.999297 25534 raft_consensus.cc:2804] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:53.997102 25559 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:54.031127 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:54.044690 22654 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 1421) took 2495 ms (client timeout 1520 ms). Trace:
I20251026 14:59:54.053346 22803 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
W20251026 14:59:54.053563 22654 rpcz_store.cc:269] 1026 14:59:51.549202 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.553511 (+ 4309us) service_pool.cc:225] Handling call
1026 14:59:54.038433 (+2484922us) spinlock_profiling.cc:243] Waited 2.15 s on lock 0x7b54001fa754. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:54.044586 (+ 6153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4954210048}
I20251026 14:59:54.054227 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.049853 23399 rpcz_store.cc:276] 1026 14:59:49.725863 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:49.730344 (+ 4481us) service_pool.cc:225] Handling call
1026 14:59:49.744333 (+ 13989us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:53.353690 (+3609357us) spinlock_profiling.cc:243] Waited 955 ms on lock 0x7b58008114d0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d10b9ffa 00007fd8d113c097 00007fd8d1154b87 00007fd8d115637e 00007fd8d11606d9 00007fd8d1160fa6 00007fd8d115d92f 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34
1026 14:59:53.983692 (+630002us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:54.006179 (+ 22487us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2195839232}
I20251026 14:59:54.055868 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.034646 25758 raft_consensus.cc:2804] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader election won for term 9
I20251026 14:59:54.057036 25758 raft_consensus.cc:697] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 9 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:54.057004 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.045749 25515 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:54.058091 25758 consensus_queue.cc:237] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:54.058120 25515 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:54.058883 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.046202 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:54.059584 23403 raft_consensus.cc:2468] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:54.000648 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:54.062465 22771 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:54.063511 25559 raft_consensus.cc:2804] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:54.063223 25522 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:54.048483 22726 raft_consensus.cc:1275] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20251026 14:59:54.034982 22659 pending_rounds.cc:85] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29: Aborting all ops after (but not including) 3
I20251026 14:59:54.051868 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.065371 22805 raft_consensus.cc:2393] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 7: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:54.066484 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.066637 25543 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:54.067288 22805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1392) took 1774 ms. Trace:
I20251026 14:59:54.067878 22805 rpcz_store.cc:276] 1026 14:59:52.292710 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.066209 (+1773499us) service_pool.cc:225] Handling call
1026 14:59:54.067192 (+ 983us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:54.018088 25755 raft_consensus.cc:697] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:54.070190 25755 consensus_queue.cc:237] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:54.032332 25598 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:54.046981 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.074668 22801 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
W20251026 14:59:54.075685 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:54.064029 25559 raft_consensus.cc:697] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:54.064664 22659 pending_rounds.cc:107] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20251026 14:59:54.078199 22659 raft_consensus.cc:2889] T 4129daa4fb70408193f07b8e6d25effe P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:54.079332 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1667) took 1841 ms. Trace:
I20251026 14:59:54.067412 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1416) took 2630 ms. Trace:
I20251026 14:59:54.069020 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.038941 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:54.039145 25193 raft_consensus.cc:2804] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 14:59:54.047078 25622 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:54.044692 25732 raft_consensus.cc:697] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 10 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:54.080941 22733 raft_consensus.cc:2410] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 12 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20251026 14:59:54.050907 25735 raft_consensus.cc:3060] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:54.034430 25702 consensus_queue.cc:1048] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.039s
I20251026 14:59:54.079797 22659 rpcz_store.cc:276] 1026 14:59:52.237869 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.013990 (+1776121us) service_pool.cc:225] Handling call
1026 14:59:54.014345 (+ 355us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:54.078984 (+ 64639us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:54.079246 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:54.085834 22728 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 10.
I20251026 14:59:54.099436 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:54.047719 25241 consensus_queue.cc:1059] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20251026 14:59:54.091092 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.080183 22652 rpcz_store.cc:276] 1026 14:59:51.436654 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.150046 (+1713392us) service_pool.cc:225] Handling call
1026 14:59:53.150518 (+ 472us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:53.171686 (+ 21168us) raft_consensus.cc:1534] Early marking committed up to index 5
1026 14:59:53.624988 (+453302us) log.cc:844] Serialized 14 byte log entry
1026 14:59:53.954236 (+329248us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:53.954739 (+ 503us) log.cc:844] Serialized 26 byte log entry
1026 14:59:53.954891 (+ 152us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:53.954924 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.954990 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:54.067182 (+112192us) raft_consensus.cc:1685] finished
1026 14:59:54.067204 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:54.067352 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18029,"prepare.queue_time_us":503578,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":42,"raft.queue_time_us":24803,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":6848,"spinlock_wait_cycles":13476736,"thread_start_us":665562,"threads_started":2,"wal-append.queue_time_us":343778}
I20251026 14:59:54.075274 25534 raft_consensus.cc:697] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [term 8 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:54.077100 25559 consensus_queue.cc:237] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:54.050107 25214 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Advancing to term 10
W20251026 14:59:54.100203 22804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1385) took 1810 ms (client timeout 2198 ms). Trace:
W20251026 14:59:54.108588 22804 rpcz_store.cc:269] 1026 14:59:52.290177 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.188040 (+897863us) service_pool.cc:225] Handling call
1026 14:59:54.100011 (+911971us) spinlock_profiling.cc:243] Waited 30.6 ms on lock 0x7b540020c1d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:54.100151 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":70326528}
I20251026 14:59:54.103724 25543 consensus_queue.cc:1048] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.006s
I20251026 14:59:54.110045 25481 raft_consensus.cc:3055] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:54.110481 25481 raft_consensus.cc:740] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:54.022904 25643 consensus_queue.cc:237] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:54.103648 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:54.080684 22805 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:54.102921 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:54.106631 22800 raft_consensus.cc:2410] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20251026 14:59:54.109892 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.114419 22804 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:54.115224 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.116004 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.116366 22804 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
I20251026 14:59:54.102733 22801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1397) took 1804 ms. Trace:
I20251026 14:59:54.104559 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:54.116938 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e49f6058b03447ca17f7a94a8b071d8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.117091 22801 rpcz_store.cc:276] 1026 14:59:52.297863 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.075532 (+1777669us) service_pool.cc:225] Handling call
1026 14:59:54.102653 (+ 27121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:54.117432 22659 raft_consensus.cc:3060] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:54.113677 22728 raft_consensus.cc:2410] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20251026 14:59:54.118047 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.118673 22801 raft_consensus.cc:2468] T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
I20251026 14:59:54.118975 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:54.119726 22728 raft_consensus.cc:2410] T 524d8339d5244ca98e86f0ff2fd02b67 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20251026 14:59:54.106372 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:54.089779 25193 raft_consensus.cc:697] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:54.119556 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.120518 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:54.121083 22728 raft_consensus.cc:2468] T 9440e0456e0648a5af21108f5fd7cfba P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:54.120815 25193 consensus_queue.cc:237] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:54.121126 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.111730 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1669) took 1899 ms. Trace:
I20251026 14:59:54.121749 22732 rpcz_store.cc:276] 1026 14:59:52.212544 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:52.212691 (+ 147us) service_pool.cc:225] Handling call
1026 14:59:52.222101 (+ 9410us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:53.613483 (+1391382us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:54.111489 (+498006us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54000b9fd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:54.111670 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2859010304}
I20251026 14:59:54.122171 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:54.122424 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.123245 22728 raft_consensus.cc:2410] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251026 14:59:54.114174 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:54.124441 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.125136 22805 raft_consensus.cc:2410] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20251026 14:59:54.124960 22801 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:54.125969 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.126058 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9440e0456e0648a5af21108f5fd7cfba" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.126616 22805 raft_consensus.cc:2468] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
I20251026 14:59:54.127159 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 11 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.127403 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:54.127805 22624 leader_election.cc:336] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.737s (SENT)
I20251026 14:59:54.128093 22805 raft_consensus.cc:2393] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 6: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:54.128237 22624 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:54.128852 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.114092 22652 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 9.
I20251026 14:59:54.129817 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.130460 22801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1395) took 1833 ms. Trace:
I20251026 14:59:54.130807 22801 rpcz_store.cc:276] 1026 14:59:52.296944 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.129590 (+1832646us) service_pool.cc:225] Handling call
1026 14:59:54.130410 (+ 820us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:54.118064 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:54.132297 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.133147 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.133370 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:54.133901 22804 raft_consensus.cc:2393] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 7: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
W20251026 14:59:54.133982 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:54.134064 22801 raft_consensus.cc:2393] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 8: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:54.134725 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.135318 22800 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 9.
I20251026 14:59:54.135339 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:54.135988 22801 raft_consensus.cc:3060] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:54.136284 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.100699 25702 consensus_queue.cc:1243] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer df3a0d1619ee48e8ab68ee0543a72a29 log is divergent from this leader: its last log entry 9.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20251026 14:59:54.140610 22733 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:54.142128 22774 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:54.144191 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:54.145323 22801 raft_consensus.cc:2468] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 10.
I20251026 14:59:54.156141 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.156844 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:59:54.158821 22799 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1398) took 1860 ms (client timeout 2189 ms). Trace:
W20251026 14:59:54.159205 22799 rpcz_store.cc:269] 1026 14:59:52.298137 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.082420 (+784283us) service_pool.cc:225] Handling call
1026 14:59:54.156685 (+1074265us) spinlock_profiling.cc:243] Waited 27.4 ms on lock 0x7b5800002a98. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8ce074451 00007fd8ce074302 00007fd8d6f0dad9 00007fd8d6090d30 00007fd8d615984b 00007fd8d61598d1 00007fd8d6107660 00007fd8d6106f44 00007fd8cf6a4834 00007fd8cf6a47c2
1026 14:59:54.158759 (+ 2074us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":63000320}
I20251026 14:59:54.160580 22799 raft_consensus.cc:1240] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20251026 14:59:54.161088 22800 raft_consensus.cc:1240] T 4e49f6058b03447ca17f7a94a8b071d8 P 0d684445287b450a8f738b6ad4b59821 [term 10 LEADER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 9. Current term is 10. Ops: []
I20251026 14:59:54.161170 22799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1488) took 2795 ms. Trace:
I20251026 14:59:54.161834 22799 rpcz_store.cc:276] 1026 14:59:51.365655 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.160009 (+2794354us) service_pool.cc:225] Handling call
1026 14:59:54.160344 (+ 335us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:54.160937 (+ 593us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:54.161116 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:54.165719 25534 consensus_queue.cc:237] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:54.161825 22800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1495) took 2518 ms. Trace:
I20251026 14:59:54.167949 22800 rpcz_store.cc:276] 1026 14:59:51.643219 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.160604 (+2517385us) service_pool.cc:225] Handling call
1026 14:59:54.160903 (+ 299us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:54.161597 (+ 694us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:54.161773 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:54.169394 22800 raft_consensus.cc:1240] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:54.170066 22800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1399) took 1871 ms. Trace:
I20251026 14:59:54.170382 22800 rpcz_store.cc:276] 1026 14:59:52.298426 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.168789 (+1870363us) service_pool.cc:225] Handling call
1026 14:59:54.169172 (+ 383us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:54.169827 (+ 655us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:54.170015 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:54.171116 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:54.096294 25622 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:54.171988 22659 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
W20251026 14:59:54.169450 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:54.178174 22774 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:54.146939 25522 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:54.152805 25543 raft_consensus.cc:2804] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20251026 14:59:54.187512 22658 raft_consensus.cc:2410] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
W20251026 14:59:54.207969 22773 leader_election.cc:336] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 2.043s (SENT)
I20251026 14:59:54.210449 22773 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:54.182387 25702 raft_consensus.cc:2804] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:54.213577 22800 raft_consensus.cc:1275] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:54.220427 25522 raft_consensus.cc:2749] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251026 14:59:54.111361 25481 consensus_queue.cc:260] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:54.244835 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:54.273144 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:54.267066 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:54.164830 25732 consensus_queue.cc:237] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:54.191466 25543 raft_consensus.cc:493] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:54.192960 25735 raft_consensus.cc:515] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:54.281069 25389 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:54.383836 22773 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 2.595s (SENT) [suppressed 82 similar messages]
W20251026 14:59:54.306104 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:54.391180 25543 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Advancing to term 11
W20251026 14:59:54.407514 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:54.212807 25563 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:54.487835 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 102647us
I20251026 14:59:54.280920 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:54.489346 22800 raft_consensus.cc:2376] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8. Re-sending same reply.
W20251026 14:59:54.491600 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:54.510627 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:54.540616 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:54.487773 25522 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:54.588356 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:54.586659 22804 raft_consensus.cc:1240] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 8. Current term is 9. Ops: []
I20251026 14:59:54.590365 22804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1483) took 3711 ms. Trace:
I20251026 14:59:54.590880 22804 rpcz_store.cc:276] 1026 14:59:50.878480 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.138616 (+3260136us) service_pool.cc:225] Handling call
1026 14:59:54.586036 (+447420us) spinlock_profiling.cc:243] Waited 16.7 ms on lock 0x7b5800002a98. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8ce074451 00007fd8ce074302 00007fd8d6f0dad9 00007fd8d6090d30 00007fd8d615984b 00007fd8d61598d1 00007fd8d611b580 00007fd8d6105ee9 00007fd8d6105992 00007fd8cf69fd34
1026 14:59:54.586382 (+ 346us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:54.590081 (+ 3699us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:54.590293 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":38485760}
I20251026 14:59:54.587368 22732 raft_consensus.cc:1240] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 2. Current term is 3. Ops: []
I20251026 14:59:54.313807 25702 raft_consensus.cc:697] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
W20251026 14:59:54.634840 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:54.605556 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:54.643199 22654 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.012s sys 0.037s
I20251026 14:59:54.380722 25214 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:54.645183 22654 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.589s user 0.012s sys 0.038s
I20251026 14:59:54.646435 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:54.659667 22697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.007s sys 0.023s
W20251026 14:59:54.660298 22697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.633s user 0.007s sys 0.023s
W20251026 14:59:54.660614 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 634020us
W20251026 14:59:54.664719 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:54.680725 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.682173 22800 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
W20251026 14:59:54.686003 22697 connection.cc:232] RPC call timeout handler was delayed by -1.0171547639999972s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15810000000000002s for a response
W20251026 14:59:54.699036 22655 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 1428) took 3147 ms (client timeout 2947 ms). Trace:
I20251026 14:59:54.699394 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:54.699916 22655 rpcz_store.cc:269] 1026 14:59:51.551195 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.697379 (+146184us) service_pool.cc:225] Handling call
1026 14:59:54.698685 (+3001306us) spinlock_profiling.cc:243] Waited 1.91 s on lock 0x7b54001fac54. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:54.698932 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4404147456}
W20251026 14:59:54.700626 22730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 1248) took 1506 ms (client timeout 1761 ms). Trace:
W20251026 14:59:54.701117 22730 rpcz_store.cc:269] 1026 14:59:53.194309 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.698952 (+1504643us) service_pool.cc:225] Handling call
1026 14:59:54.700550 (+ 1598us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:54.702311 22657 raft_consensus.cc:1275] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20251026 14:59:54.702808 22730 raft_consensus.cc:1240] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20251026 14:59:54.703799 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:54.705754 22807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44582 (request call id 1402) took 2406 ms (client timeout 1581 ms). Trace:
W20251026 14:59:54.706652 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:54.706314 22807 rpcz_store.cc:269] 1026 14:59:52.299272 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:52.528776 (+229504us) service_pool.cc:225] Handling call
1026 14:59:54.705494 (+2176718us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b5400389fd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:54.705663 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":560307456}
I20251026 14:59:54.706164 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:54.707719 22655 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:54.707811 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.708786 22807 raft_consensus.cc:2393] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 7: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:54.704507 22730 raft_consensus.cc:1240] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 8. Current term is 9. Ops: [6.5-8.6]
I20251026 14:59:54.709776 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.711112 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.712579 22807 raft_consensus.cc:1240] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 8. Current term is 9. Ops: []
I20251026 14:59:54.713897 22808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1361) took 4638 ms. Trace:
I20251026 14:59:54.718787 22808 rpcz_store.cc:276] 1026 14:59:50.075009 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:52.942520 (+2867511us) service_pool.cc:225] Handling call
1026 14:59:52.942875 (+ 355us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:52.958194 (+ 15319us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:52.958240 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:52.958312 (+ 72us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:52.958366 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:54.713612 (+1755246us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b5400333bd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:54.713677 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:54.713834 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":34593536}
I20251026 14:59:54.714584 22802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1373) took 4438 ms. Trace:
I20251026 14:59:54.721582 22802 rpcz_store.cc:276] 1026 14:59:50.276167 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.867535 (+1591368us) service_pool.cc:225] Handling call
1026 14:59:51.877003 (+ 9468us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:53.933671 (+2056668us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:54.714369 (+780698us) spinlock_profiling.cc:243] Waited 2 s on lock 0x7b540004e2d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:54.714532 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4606956288}
I20251026 14:59:54.576876 25563 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:54.724934 25715 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.009s sys 0.006s
I20251026 14:59:54.595392 25522 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:54.741505 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.738118 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.937691 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.938120 22800 raft_consensus.cc:2468] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
W20251026 14:59:54.914912 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:54.941987 22774 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:54.942175 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.945613 22806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1367) took 4868 ms. Trace:
I20251026 14:59:54.946012 22806 rpcz_store.cc:276] 1026 14:59:50.076777 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.843354 (+1766577us) service_pool.cc:225] Handling call
1026 14:59:51.843706 (+ 352us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:53.735599 (+1891893us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:54.945371 (+1209772us) spinlock_profiling.cc:243] Waited 1.88 s on lock 0x7b54001b85d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:54.945553 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4327354112}
I20251026 14:59:54.713253 22807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1496) took 1001 ms. Trace:
I20251026 14:59:54.947022 22807 rpcz_store.cc:276] 1026 14:59:53.712167 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.712039 (+999872us) service_pool.cc:225] Handling call
1026 14:59:54.712347 (+ 308us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:54.712997 (+ 650us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:54.713193 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:54.951864 25576 leader_election.cc:290] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:54.920475 25786 log.cc:927] Time spent T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20251026 14:59:54.952494 24740 raft_consensus.cc:2804] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:54.951776 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.952994 25583 raft_consensus.cc:3055] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:54.921718 25214 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:54.953392 22804 raft_consensus.cc:2468] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
I20251026 14:59:54.953506 25583 raft_consensus.cc:740] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:54.954515 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:54.954936 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:54.955624 22801 raft_consensus.cc:2368] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 9. Current term is 10.
I20251026 14:59:54.956418 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:54.920625 25735 leader_election.cc:290] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Requested vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
W20251026 14:59:54.901656 25670 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.745s user 0.001s sys 0.030s
W20251026 14:59:54.964162 25670 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.808s user 0.001s sys 0.033s
I20251026 14:59:54.964377 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.953191 24740 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:54.965567 24740 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:54.965950 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.967864 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 11 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:54.969571 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:54.922127 25783 raft_consensus.cc:493] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:54.970485 25783 raft_consensus.cc:515] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:54.957556 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.952526 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:54.971947 22808 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:54.957005 22806 raft_consensus.cc:1275] T dea6ba9f8ecc4124bfedb6febafde72b P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:54.955606 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:59:54.926977 25598 consensus_meta.cc:287] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e: Time spent flushing consensus metadata: real 0.846s user 0.007s sys 0.005s
I20251026 14:59:54.972839 22806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1511) took 1141 ms. Trace:
I20251026 14:59:54.954615 25583 consensus_queue.cc:260] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:54.684481 25586 raft_consensus.cc:493] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:54.929710 25622 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:54.926313 25515 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:54.932061 23400 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40908 (request call id 155) took 3117 ms (client timeout 1730 ms). Trace:
W20251026 14:59:54.937736 25715 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.780s user 0.013s sys 0.009s
W20251026 14:59:54.807343 25529 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.626s user 0.029s sys 0.070s
W20251026 14:59:54.940261 25534 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.717s user 0.009s sys 0.005s
W20251026 14:59:54.985773 25534 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.763s user 0.010s sys 0.006s
W20251026 14:59:54.940492 25755 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.656s user 0.003s sys 0.012s
W20251026 14:59:54.986661 25755 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.702s user 0.004s sys 0.012s
W20251026 14:59:54.943362 25643 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.719s user 0.001s sys 0.014s
I20251026 14:59:54.933768 25185 leader_election.cc:290] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:54.976203 25598 raft_consensus.cc:2749] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251026 14:59:54.988137 25643 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.764s user 0.001s sys 0.015s
I20251026 14:59:54.808961 25702 consensus_queue.cc:237] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:54.982039 25523 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:54.977846 25586 raft_consensus.cc:515] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:54.858568 25196 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:54.806818 25543 raft_consensus.cc:515] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:54.984823 25529 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.830s user 0.035s sys 0.086s
W20251026 14:59:54.979087 25732 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.007s sys 0.021s
I20251026 14:59:54.993320 25583 raft_consensus.cc:3060] T 7de11d91eebd4cd0b3183e4c044d4004 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:54.991887 25737 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:55.017422 25732 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s user 0.008s sys 0.022s
W20251026 14:59:55.058414 25774 log.cc:927] Time spent T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 1.125s user 0.005s sys 0.010s
W20251026 14:59:55.100134 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:55.101550 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:55.101606 22650 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.719s user 0.002s sys 0.010s
W20251026 14:59:55.101899 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:55.102267 22650 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.720s user 0.002s sys 0.010s
W20251026 14:59:55.102550 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:55.103433 22622 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:55.104367 22650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1669) took 2864 ms. Trace:
W20251026 14:59:54.726475 22624 leader_election.cc:336] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.505s (SENT)
I20251026 14:59:55.104979 22650 rpcz_store.cc:276] 1026 14:59:52.240305 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.028254 (+1787949us) service_pool.cc:225] Handling call
1026 14:59:54.028588 (+ 334us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:54.028954 (+ 366us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:54.028983 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:54.045448 (+ 16465us) log.cc:844] Serialized 26 byte log entry
1026 14:59:54.382287 (+336839us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:55.103033 (+720746us) log.cc:844] Serialized 14 byte log entry
1026 14:59:55.103660 (+ 627us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.103793 (+ 133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:55.103914 (+ 121us) raft_consensus.cc:1685] finished
1026 14:59:55.103946 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:55.104225 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":677376,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":87,"raft.queue_time_us":108,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":7163,"thread_start_us":1009443,"threads_started":2,"wal-append.queue_time_us":737133}
W20251026 14:59:55.106837 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 380231us
W20251026 14:59:55.110725 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:54.725786 22652 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.007s sys 0.009s
W20251026 14:59:55.111465 22659 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.937s user 0.012s sys 0.000s
W20251026 14:59:55.111632 22652 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.980s user 0.008s sys 0.010s
W20251026 14:59:55.111976 22659 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.938s user 0.012s sys 0.000s
W20251026 14:59:55.112643 22624 leader_election.cc:336] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.582s (SENT)
I20251026 14:59:55.113152 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1672) took 2809 ms. Trace:
I20251026 14:59:55.113509 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1673) took 2749 ms. Trace:
I20251026 14:59:55.113651 22652 rpcz_store.cc:276] 1026 14:59:52.304202 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.130050 (+1825848us) service_pool.cc:225] Handling call
1026 14:59:54.130362 (+ 312us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:54.130721 (+ 359us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:54.130758 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:54.131333 (+ 575us) log.cc:844] Serialized 26 byte log entry
1026 14:59:55.112448 (+981115us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:55.112609 (+ 161us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.112736 (+ 127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:55.112783 (+ 47us) raft_consensus.cc:1685] finished
1026 14:59:55.112798 (+ 15us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:55.113062 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":237,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":147,"thread_start_us":980407,"threads_started":1,"wal-append.queue_time_us":883926}
I20251026 14:59:55.114008 22659 rpcz_store.cc:276] 1026 14:59:52.364287 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.172683 (+1808396us) service_pool.cc:225] Handling call
1026 14:59:54.172983 (+ 300us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:54.173344 (+ 361us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:54.173378 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:54.173826 (+ 448us) log.cc:844] Serialized 26 byte log entry
1026 14:59:55.112885 (+939059us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:55.113012 (+ 127us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.113109 (+ 97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:55.113178 (+ 69us) raft_consensus.cc:1685] finished
1026 14:59:55.113189 (+ 11us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:55.113433 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":213,"raft.queue_time_us":7962,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":4061,"thread_start_us":938396,"threads_started":1,"wal-append.queue_time_us":872086}
I20251026 14:59:55.117607 22659 pending_rounds.cc:85] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29: Aborting all ops after (but not including) 2
I20251026 14:59:55.118003 22659 pending_rounds.cc:107] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20251026 14:59:55.118427 22655 raft_consensus.cc:1240] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
W20251026 14:59:54.789965 22728 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.664s user 0.007s sys 0.022s
W20251026 14:59:55.121521 22729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.951s user 0.029s sys 0.062s
W20251026 14:59:55.122267 22729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.952s user 0.029s sys 0.062s
W20251026 14:59:55.122521 22725 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.979s user 0.014s sys 0.017s
W20251026 14:59:55.123039 22725 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.980s user 0.014s sys 0.017s
I20251026 14:59:54.760941 22730 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:54.853778 22732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47996 (request call id 190) took 1207 ms. Trace:
I20251026 14:59:55.125078 22725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47996 (request call id 188) took 1490 ms. Trace:
I20251026 14:59:55.126212 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:55.128805 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:55.131446 22800 raft_consensus.cc:1240] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 9. Current term is 10. Ops: []
I20251026 14:59:55.124825 22732 rpcz_store.cc:276] 1026 14:59:53.646239 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.123574 (+477335us) service_pool.cc:225] Handling call
1026 14:59:54.123845 (+ 271us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:54.605015 (+481170us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:54.853430 (+248415us) spinlock_profiling.cc:243] Waited 463 ms on lock 0x7b54002dbc54. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:54.853687 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1065154432}
I20251026 14:59:55.125628 22725 rpcz_store.cc:276] 1026 14:59:53.645310 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.970346 (+325036us) service_pool.cc:225] Handling call
1026 14:59:53.970625 (+ 279us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:53.970817 (+ 192us) raft_consensus.cc:1534] Early marking committed up to index 7
1026 14:59:53.992908 (+ 22091us) log.cc:844] Serialized 14 byte log entry
1026 14:59:55.123692 (+1130784us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:55.123816 (+ 124us) raft_consensus.cc:1645] Marking committed up to 7
1026 14:59:55.123956 (+ 140us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.124681 (+ 725us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:55.124958 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":21949,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":679730,"raft.run_cpu_time_us":66,"raft.run_wall_time_us":66,"spinlock_wait_cycles":999168,"thread_start_us":1129687,"threads_started":2,"wal-append.queue_time_us":288695}
I20251026 14:59:55.133921 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:55.135159 22801 raft_consensus.cc:2393] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 9: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
W20251026 14:59:55.122119 22728 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.996s user 0.009s sys 0.023s
I20251026 14:59:55.135246 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:55.136500 22725 raft_consensus.cc:2410] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20251026 14:59:55.135317 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:55.138404 22732 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:55.138725 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 11 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:54.973450 22806 rpcz_store.cc:276] 1026 14:59:53.831273 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.946884 (+1115611us) service_pool.cc:225] Handling call
1026 14:59:54.947206 (+ 322us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:54.972532 (+ 25326us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:54.972769 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:54.974817 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:55.140544 22807 raft_consensus.cc:1240] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 11 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 8. Current term is 11. Ops: []
W20251026 14:59:54.978946 23400 rpcz_store.cc:269] 1026 14:59:51.814920 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.817999 (+ 3079us) service_pool.cc:225] Handling call
1026 14:59:54.931868 (+3113869us) spinlock_profiling.cc:243] Waited 1.89 s on lock 0x7b5400460d54. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:54.931988 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4346873216}
I20251026 14:59:55.116714 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:55.118490 22659 raft_consensus.cc:2889] T 524d8339d5244ca98e86f0ff2fd02b67 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:55.157999 22800 raft_consensus.cc:1240] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 8. Current term is 9. Ops: []
I20251026 14:59:55.156982 25543 leader_election.cc:290] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 11 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:54.997809 25196 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:55.182085 22725 raft_consensus.cc:2393] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 11: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
W20251026 14:59:55.189430 22699 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.605s user 0.007s sys 0.000s
W20251026 14:59:55.190040 22699 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.660s user 0.008s sys 0.000s
I20251026 14:59:55.190855 25586 leader_election.cc:290] T 129989aafdc84d6b8e56fe97c7182fff P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:55.059901 25783 leader_election.cc:290] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:55.194657 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:55.196447 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54564 (request call id 1431) took 2994 ms. Trace:
I20251026 14:59:55.201586 25804 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df3a0d1619ee48e8ab68ee0543a72a29)
I20251026 14:59:55.197038 22651 rpcz_store.cc:276] 1026 14:59:52.201508 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.985914 (+1784406us) service_pool.cc:225] Handling call
1026 14:59:54.027337 (+ 41423us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:54.033084 (+ 5747us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.195918 (+1162834us) spinlock_profiling.cc:243] Waited 41 ms on lock 0x7b540010dbd0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:55.196199 (+ 281us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":94195584}
I20251026 14:59:55.204957 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:55.207789 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:55.207899 25604 raft_consensus.cc:2749] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20251026 14:59:55.210753 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:55.208462 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 90189.2us
I20251026 14:59:55.192936 24740 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:55.214566 24740 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:55.215165 22657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1650) took 3375 ms. Trace:
I20251026 14:59:55.215734 22657 rpcz_store.cc:276] 1026 14:59:51.840038 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.727828 (+1887790us) service_pool.cc:225] Handling call
1026 14:59:54.701856 (+974028us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:54.702821 (+ 965us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.208333 (+505512us) spinlock_profiling.cc:243] Waited 961 ms on lock 0x7b54001fac50. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115d44d 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:55.214949 (+ 6616us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2211416832}
I20251026 14:59:55.218066 22652 raft_consensus.cc:2468] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 8.
I20251026 14:59:55.206126 25576 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:55.207932 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:55.221058 23402 raft_consensus.cc:2376] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 5. Re-sending same reply.
I20251026 14:59:55.220868 25576 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:55.223560 22622 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:55.224512 22732 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
I20251026 14:59:55.226567 25602 raft_consensus.cc:2804] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:55.227452 25602 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:55.228050 25602 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:55.228611 22727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35092 (request call id 1686) took 2941 ms. Trace:
I20251026 14:59:55.229058 22727 rpcz_store.cc:276] 1026 14:59:52.286825 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:52.419402 (+132577us) service_pool.cc:225] Handling call
1026 14:59:52.434217 (+ 14815us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:53.465166 (+1030949us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.227959 (+1762793us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54000b90d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:55.228379 (+ 420us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2369415424}
I20251026 14:59:55.230547 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:55.230690 22622 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:55.232041 25624 raft_consensus.cc:2804] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251026 14:59:55.232163 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:55.232553 22622 leader_election.cc:304] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:55.232579 25624 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:55.234375 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:55.236585 22805 raft_consensus.cc:1240] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 8. Current term is 9. Ops: []
I20251026 14:59:55.237342 22805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1487) took 4357 ms. Trace:
I20251026 14:59:55.240293 25522 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:55.241849 22729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1272) took 1407 ms. Trace:
I20251026 14:59:55.242455 22729 rpcz_store.cc:276] 1026 14:59:53.834837 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.150877 (+316040us) service_pool.cc:225] Handling call
1026 14:59:54.169704 (+ 18827us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:54.169958 (+ 254us) raft_consensus.cc:1534] Early marking committed up to index 6
1026 14:59:54.170323 (+ 365us) log.cc:844] Serialized 14 byte log entry
1026 14:59:55.207035 (+1036712us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:55.234524 (+ 27489us) raft_consensus.cc:1645] Marking committed up to 6
1026 14:59:55.241409 (+ 6885us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.241543 (+ 134us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:55.241779 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42471,"prepare.queue_time_us":1283,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":1964,"raft.queue_time_us":111199,"raft.run_cpu_time_us":351,"raft.run_wall_time_us":350,"thread_start_us":952303,"threads_started":1,"wal-append.queue_time_us":847484}
I20251026 14:59:55.245297 22808 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 8.
I20251026 14:59:55.253312 22651 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:55.237722 22805 rpcz_store.cc:276] 1026 14:59:50.879483 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.144790 (+3265307us) service_pool.cc:225] Handling call
1026 14:59:55.235673 (+1090883us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b5800002a98. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8ce074451 00007fd8ce074302 00007fd8d6f0dad9 00007fd8d6090d30 00007fd8d615984b 00007fd8d61598d1 00007fd8d611b580 00007fd8d6105ee9 00007fd8d6105992 00007fd8cf69fd34
1026 14:59:55.236064 (+ 391us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:55.237052 (+ 988us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.237250 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24674688}
I20251026 14:59:55.233383 25624 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:55.223591 25576 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:55.262550 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 60484.8us
I20251026 14:59:55.261695 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:55.234930 25586 raft_consensus.cc:493] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:55.266206 22725 raft_consensus.cc:2410] T 3103efcd5601429a95eeb7a270f6fab1 P 562bb1e1c8e445b8baecd29c3669658e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 12 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20251026 14:59:55.266511 22727 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:55.266598 25804 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:55.265034 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:55.270812 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:55.274709 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:53.759234 22656 rpcz_store.cc:276] 1026 14:59:51.330262 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.087889 (+1757627us) service_pool.cc:225] Handling call
1026 14:59:53.088502 (+ 613us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:53.088783 (+ 281us) raft_consensus.cc:1534] Early marking committed up to index 4
1026 14:59:53.416525 (+327742us) log.cc:844] Serialized 14 byte log entry
1026 14:59:53.743158 (+326633us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:53.743214 (+ 56us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:53.743280 (+ 66us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:53.743345 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:53.743465 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":526862,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":84,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":7083,"thread_start_us":624766,"threads_started":2}
I20251026 14:59:55.282033 25801 consensus_queue.cc:1048] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20251026 14:59:55.281142 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:55.284587 22656 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 9.
I20251026 14:59:55.281085 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:55.286407 22728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1269) took 1543 ms. Trace:
I20251026 14:59:55.281154 25636 leader_election.cc:290] T 45fb267b6944430e8237d9baad843789 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:55.299556 25576 raft_consensus.cc:2804] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251026 14:59:55.311448 25196 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:55.313676 23370 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:55.313858 22697 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:55.319235 22651 raft_consensus.cc:3060] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Advancing to term 10
W20251026 14:59:55.327203 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 94814.6us
I20251026 14:59:55.329680 22622 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:55.329974 22799 raft_consensus.cc:1240] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20251026 14:59:55.331175 22622 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:55.331279 25819 raft_consensus.cc:2749] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251026 14:59:55.332476 25819 raft_consensus.cc:2804] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:55.344427 22732 raft_consensus.cc:2393] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 9: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
W20251026 14:59:55.320688 22624 leader_election.cc:336] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.642s (SENT)
I20251026 14:59:55.349474 22624 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:55.322103 25576 raft_consensus.cc:493] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
W20251026 14:59:55.334642 25753 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.334s user 0.028s sys 0.007s
I20251026 14:59:55.295056 22728 rpcz_store.cc:276] 1026 14:59:53.743331 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.123906 (+380575us) service_pool.cc:225] Handling call
1026 14:59:54.124267 (+ 361us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:54.124675 (+ 408us) raft_consensus.cc:1534] Early marking committed up to index 4
1026 14:59:54.125352 (+ 677us) log.cc:844] Serialized 14 byte log entry
1026 14:59:55.209475 (+1084123us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:55.240365 (+ 30890us) log.cc:844] Serialized 26 byte log entry
1026 14:59:55.240706 (+ 341us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:55.241222 (+ 516us) log.cc:844] Serialized 14 byte log entry
1026 14:59:55.271609 (+ 30387us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.273876 (+ 2267us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:55.273908 (+ 32us) raft_consensus.cc:1685] finished
1026 14:59:55.273920 (+ 12us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:55.286342 (+ 12422us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":70908,"prepare.queue_time_us":3707,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":89,"raft.queue_time_us":830287,"raft.run_cpu_time_us":427,"raft.run_wall_time_us":3387,"thread_start_us":1010173,"threads_started":1,"wal-append.queue_time_us":911385}
I20251026 14:59:55.353684 22657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1678) took 2180 ms. Trace:
I20251026 14:59:55.354162 22657 rpcz_store.cc:276] 1026 14:59:53.170633 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:55.217031 (+2046398us) service_pool.cc:225] Handling call
1026 14:59:55.272564 (+ 55533us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:55.346725 (+ 74161us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:55.346752 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:55.350354 (+ 3602us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:55.350410 (+ 56us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.350476 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:55.350608 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:55.355683 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:55.356554 22657 raft_consensus.cc:2468] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 5.
W20251026 14:59:55.357470 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:55.357782 25804 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
W20251026 14:59:55.351990 25807 log.cc:927] Time spent T 5b0536aab3b541a2a846d187e111c09c P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.082s user 0.002s sys 0.000s
I20251026 14:59:55.361008 25576 raft_consensus.cc:3060] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:55.363199 22725 raft_consensus.cc:2393] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 10: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:55.364233 25621 raft_consensus.cc:3060] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:55.367997 23402 raft_consensus.cc:1275] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20251026 14:59:55.369726 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
W20251026 14:59:55.371094 22773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:55.371021 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:55.372224 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:55.372948 22728 raft_consensus.cc:2468] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
W20251026 14:59:55.375777 25753 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.476s user 0.029s sys 0.007s
I20251026 14:59:55.378301 22654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1671) took 3136 ms. Trace:
I20251026 14:59:55.380263 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:55.380172 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:55.381531 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 68329.5us
I20251026 14:59:55.381841 25670 raft_consensus.cc:2804] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251026 14:59:55.364341 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:55.378978 22654 rpcz_store.cc:276] 1026 14:59:52.241722 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.054694 (+1812972us) service_pool.cc:225] Handling call
1026 14:59:54.055016 (+ 322us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:54.055482 (+ 466us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:54.055523 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:54.056163 (+ 640us) log.cc:844] Serialized 52 byte log entry
1026 14:59:54.650701 (+594538us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:55.271164 (+620463us) log.cc:844] Serialized 14 byte log entry
1026 14:59:55.377119 (+105955us) log.cc:844] Serialized 14 byte log entry
1026 14:59:55.377578 (+ 459us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.377684 (+ 106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:55.377749 (+ 65us) raft_consensus.cc:1685] finished
1026 14:59:55.377769 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:55.378036 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18956,"prepare.queue_time_us":414965,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":31681,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":290,"spinlock_wait_cycles":1792,"thread_start_us":1077594,"threads_started":2,"wal-append.queue_time_us":736930}
I20251026 14:59:55.388229 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:55.388458 25186 consensus_queue.cc:1048] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20251026 14:59:55.388803 22733 raft_consensus.cc:1240] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [term 9 LEADER]: Rejecting Update request from peer df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20251026 14:59:55.390728 25804 consensus_queue.cc:1048] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251026 14:59:55.390959 25602 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:55.392998 22804 raft_consensus.cc:1240] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:55.394289 25670 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:55.394811 25670 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:55.397739 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:55.359896 25624 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:55.401197 25624 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:55.401733 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:55.402336 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3620632bd772435f97f799be96d493b8" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:55.402510 22658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54564 (request call id 1409) took 4052 ms (client timeout 3537 ms). Trace:
I20251026 14:59:55.402793 25783 raft_consensus.cc:2804] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Leader pre-election won for term 6
W20251026 14:59:55.402956 22658 rpcz_store.cc:269] 1026 14:59:51.350200 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:51.715326 (+365126us) service_pool.cc:225] Handling call
1026 14:59:55.402038 (+3686712us) spinlock_profiling.cc:243] Waited 2.47 s on lock 0x7b5400241154. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:55.402301 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5682461568}
W20251026 14:59:55.404392 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 90737.6us
I20251026 14:59:55.403343 25783 raft_consensus.cc:493] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:55.406103 22802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1510) took 1591 ms. Trace:
I20251026 14:59:55.343694 25819 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:55.410356 22650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1675) took 2841 ms. Trace:
I20251026 14:59:55.318285 25586 raft_consensus.cc:515] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:55.437094 22726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1247) took 2580 ms. Trace:
I20251026 14:59:55.437736 22726 rpcz_store.cc:276] 1026 14:59:52.856816 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.031062 (+1174246us) service_pool.cc:225] Handling call
1026 14:59:54.031352 (+ 290us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:54.064423 (+ 33071us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.410095 (+1345672us) spinlock_profiling.cc:243] Waited 16.8 ms on lock 0x7b54002dcb54. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:55.436849 (+ 26754us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":38586624}
I20251026 14:59:55.439015 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:55.406427 22697 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:55.441498 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba8784cbec044fb7b5b098961f0f94b2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:55.420418 22622 leader_election.cc:304] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:55.393115 25602 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 election: Requested vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
W20251026 14:59:55.436028 22653 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32888 (request call id 157) took 3359 ms (client timeout 1728 ms). Trace:
I20251026 14:59:55.409802 25783 raft_consensus.cc:3060] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:55.419127 22650 rpcz_store.cc:276] 1026 14:59:52.565948 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:55.108389 (+2542441us) service_pool.cc:225] Handling call
1026 14:59:55.109167 (+ 778us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:55.110236 (+ 1069us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:55.110268 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:55.119107 (+ 8839us) log.cc:844] Serialized 26 byte log entry
1026 14:59:55.298941 (+179834us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:55.333919 (+ 34978us) log.cc:844] Serialized 14 byte log entry
1026 14:59:55.334065 (+ 146us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.334115 (+ 50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:55.407115 (+ 73000us) raft_consensus.cc:1685] finished
1026 14:59:55.407141 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:55.407317 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":86510,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":61,"raft.queue_time_us":58879,"raft.run_cpu_time_us":537,"raft.run_wall_time_us":3385,"thread_start_us":179092,"threads_started":1,"wal-append.queue_time_us":184267}
I20251026 14:59:55.454082 22726 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
W20251026 14:59:55.451572 22653 rpcz_store.cc:269] 1026 14:59:52.076391 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:52.154236 (+ 77845us) service_pool.cc:225] Handling call
1026 14:59:55.435501 (+3281265us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b540012ded4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:55.435753 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3865323776}
I20251026 14:59:55.456836 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e923fc1e1fa340a7b482357487981b39" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:55.457433 22651 raft_consensus.cc:2468] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 10.
I20251026 14:59:55.417085 25636 raft_consensus.cc:2749] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251026 14:59:55.459740 22731 raft_consensus.cc:2376] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9. Re-sending same reply.
I20251026 14:59:55.464823 23367 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:55.464679 22554 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50170 (request call id 62) took 12675 ms (client timeout 15000 ms). Trace:
I20251026 14:59:55.421067 25819 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:55.478238 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:55.439221 25586 leader_election.cc:290] T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:55.484205 22725 raft_consensus.cc:2468] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:55.408979 22802 rpcz_store.cc:276] 1026 14:59:53.814127 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.722697 (+908570us) service_pool.cc:225] Handling call
1026 14:59:55.225334 (+502637us) spinlock_profiling.cc:243] Waited 2.24 ms on lock 0x7b5800002a98. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8cdef2381 00007fd8ce074451 00007fd8ce074302 00007fd8d6f0dad9 00007fd8d6090d30 00007fd8d615984b 00007fd8d61598d1 00007fd8d611b580 00007fd8d6105ee9 00007fd8d6105992 00007fd8cf69fd34
1026 14:59:55.225757 (+ 423us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:55.226462 (+ 705us) raft_consensus.cc:1534] Early marking committed up to index 5
1026 14:59:55.226960 (+ 498us) log.cc:844] Serialized 14 byte log entry
1026 14:59:55.405533 (+178573us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:55.405592 (+ 59us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:55.405647 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.405719 (+ 72us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:55.405902 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":79728,"prepare.queue_time_us":45205,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":6792,"raft.queue_time_us":79864,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":132,"spinlock_wait_cycles":5144960,"thread_start_us":98187,"threads_started":1,"wal-append.queue_time_us":98624}
I20251026 14:59:55.486791 22726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:55.497759 25586 raft_consensus.cc:2804] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251026 14:59:55.498445 25586 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:55.498970 25586 raft_consensus.cc:3060] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:55.502578 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:55.503582 22729 raft_consensus.cc:2410] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20251026 14:59:55.504456 22729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6a1f6f52f31d4a9e98b2c86f2358ea8b" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:55.508378 25481 raft_consensus.cc:3060] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Advancing to term 6
W20251026 14:59:55.509260 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 195996us
I20251026 14:59:55.512720 22807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44594 (request call id 1512) took 1678 ms. Trace:
W20251026 14:59:55.512912 22624 leader_election.cc:336] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.664s (SENT)
I20251026 14:59:55.513152 22807 rpcz_store.cc:276] 1026 14:59:53.834553 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.947900 (+1113347us) service_pool.cc:225] Handling call
1026 14:59:54.965092 (+ 17192us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:55.176848 (+211756us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.512400 (+335552us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b5400334fd4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:55.512647 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":400829312}
I20251026 14:59:55.519119 25527 raft_consensus.cc:2749] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:55.519670 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:55.520110 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:55.520821 22800 raft_consensus.cc:2368] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 10. Current term is 11.
I20251026 14:59:55.520951 23402 raft_consensus.cc:1275] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20251026 14:59:55.521549 25196 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:55.522692 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:55.523592 22802 raft_consensus.cc:2393] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 7: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:55.515125 23370 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:55.525465 25527 raft_consensus.cc:2804] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20251026 14:59:55.525861 25527 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:55.526212 25527 raft_consensus.cc:3060] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:55.532428 25563 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:55.498564 22654 raft_consensus.cc:1240] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 6. Current term is 7. Ops: []
I20251026 14:59:55.559181 22654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32888 (request call id 159) took 1568 ms. Trace:
I20251026 14:59:55.559489 22654 rpcz_store.cc:276] 1026 14:59:53.990784 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:55.427906 (+1437122us) service_pool.cc:225] Handling call
1026 14:59:55.498374 (+ 70468us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:55.558956 (+ 60582us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.559135 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251026 14:59:55.491935 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 159351us
I20251026 14:59:55.533890 22734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 1262) took 1842 ms. Trace:
I20251026 14:59:55.534816 22655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1674) took 2969 ms. Trace:
I20251026 14:59:55.535238 22653 pending_rounds.cc:85] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29: Aborting all ops after (but not including) 5
I20251026 14:59:55.560874 22653 pending_rounds.cc:107] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20251026 14:59:55.535955 21512 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 14:59:55.560962 22624 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:55.537099 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 72531.1us
I20251026 14:59:55.516950 22729 raft_consensus.cc:2393] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 8: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:55.543207 22725 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37bde08107a44bac947149e44937c2e6" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:55.517236 22728 raft_consensus.cc:2468] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 6.
I20251026 14:59:55.544384 25709 consensus_queue.cc:1048] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251026 14:59:55.545624 22657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1691) took 1642 ms. Trace:
I20251026 14:59:55.546669 22652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1681) took 2373 ms. Trace:
I20251026 14:59:55.564035 22652 rpcz_store.cc:276] 1026 14:59:53.173182 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:55.219063 (+2045881us) service_pool.cc:225] Handling call
1026 14:59:55.219688 (+ 625us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:55.256110 (+ 36422us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:55.256176 (+ 66us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:55.261630 (+ 5454us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:55.261688 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.546387 (+284699us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b54001d97d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:55.546447 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:55.546609 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31821696}
W20251026 14:59:55.541812 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 61587.3us
I20251026 14:59:55.541958 22799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44582 (request call id 1387) took 3251 ms. Trace:
W20251026 14:59:55.544435 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 138189us
I20251026 14:59:55.541812 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:55.553423 22731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "955c860c7f6f4f81bbe8e59e4c3dbcf5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:55.553623 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f688f64ce4e84d419d88768435bf773f" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:55.554843 22733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35084 (request call id 1271) took 1734 ms. Trace:
I20251026 14:59:55.556829 22726 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:55.537053 22727 raft_consensus.cc:2468] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:55.561141 22653 raft_consensus.cc:2889] T 4e49f6058b03447ca17f7a94a8b071d8 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:55.566576 22799 rpcz_store.cc:276] 1026 14:59:52.290945 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.162879 (+1871934us) service_pool.cc:225] Handling call
1026 14:59:54.163170 (+ 291us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:55.330466 (+1167296us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.541646 (+211180us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b540038a9d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:55.541895 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2682146560}
I20251026 14:59:55.560503 22655 rpcz_store.cc:276] 1026 14:59:52.565175 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.708732 (+2143557us) service_pool.cc:225] Handling call
1026 14:59:54.709099 (+ 367us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:55.119345 (+410246us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.534502 (+415157us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b54000dcd54. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:55.534752 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":938849024}
I20251026 14:59:55.563871 22657 rpcz_store.cc:276] 1026 14:59:53.903208 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:55.403377 (+1500169us) service_pool.cc:225] Handling call
1026 14:59:55.403863 (+ 486us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:55.404679 (+ 816us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:55.404709 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:55.405449 (+ 740us) log.cc:844] Serialized 26 byte log entry
1026 14:59:55.426091 (+ 20642us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:55.426175 (+ 84us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.426280 (+ 105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:55.545345 (+119065us) raft_consensus.cc:1685] finished
1026 14:59:55.545376 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:55.545535 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2310,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":43017,"thread_start_us":19567,"threads_started":1,"wal-append.queue_time_us":135486}
I20251026 14:59:55.560421 22734 rpcz_store.cc:276] 1026 14:59:53.691835 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:53.954379 (+262544us) service_pool.cc:225] Handling call
1026 14:59:55.533555 (+1579176us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54001e96d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:55.533705 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2650361600}
I20251026 14:59:55.567996 22733 rpcz_store.cc:276] 1026 14:59:53.820360 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.151973 (+331613us) service_pool.cc:225] Handling call
1026 14:59:54.152509 (+ 536us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:55.389514 (+1237005us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.554446 (+164932us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54002dad54. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:55.554768 (+ 322us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2841340672}
I20251026 14:59:55.579977 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:55.584831 22731 raft_consensus.cc:2393] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 8: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:55.586727 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dea6ba9f8ecc4124bfedb6febafde72b" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:55.588409 22732 raft_consensus.cc:2468] T f688f64ce4e84d419d88768435bf773f P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
I20251026 14:59:55.590272 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9423e278399742f48ca9ee203d7f92cb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:55.592386 25193 raft_consensus.cc:493] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:55.593098 25193 raft_consensus.cc:515] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:55.593747 22728 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e6d661a639541b3a40bf3c1fc6c0a88" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:55.593974 22622 leader_election.cc:304] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:55.585575 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:55.594734 25515 consensus_queue.cc:1059] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.015s
I20251026 14:59:55.595651 22655 raft_consensus.cc:2393] T 11ae57d90c614ce2993629f95b777120 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e in current term 10: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:55.595893 25755 raft_consensus.cc:2749] T 59b48d4137e3417983741d9cc5f92f42 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20251026 14:59:55.584285 22773 leader_election.cc:336] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 3.198s (SENT)
I20251026 14:59:55.595274 25193 leader_election.cc:290] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:55.598367 22773 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:55.611676 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d267e50950c4ef2b95c745e7103cafb" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:55.614771 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:55.595728 25563 raft_consensus.cc:2749] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251026 14:59:55.623853 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:55.629489 25702 raft_consensus.cc:2749] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251026 14:59:55.630580 22697 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:55.480284 22554 rpcz_store.cc:269] 1026 14:59:42.789034 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:42.789389 (+ 355us) service_pool.cc:225] Handling call
1026 14:59:55.464318 (+12674929us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 14:59:48.249631 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 14:59:48.293798 (+ 44167us) write_op.cc:432] Acquiring schema lock in shared mode
1026 14:59:48.293838 (+ 40us) write_op.cc:435] Acquired schema lock
1026 14:59:48.293856 (+ 18us) tablet.cc:660] Decoding operations
1026 14:59:48.294330 (+ 474us) write_op.cc:620] Acquiring the partition lock for write op
1026 14:59:48.294392 (+ 62us) write_op.cc:641] Partition lock acquired for write op
1026 14:59:48.294419 (+ 27us) tablet.cc:683] Acquiring locks for 2 operations
1026 14:59:48.332359 (+ 37940us) tablet.cc:699] Row locks acquired
1026 14:59:48.332381 (+ 22us) write_op.cc:260] PREPARE: finished
1026 14:59:48.332534 (+ 153us) write_op.cc:270] Start()
1026 14:59:48.332654 (+ 120us) write_op.cc:276] Timestamp: P: 1761490788332484 usec, L: 0
1026 14:59:48.332674 (+ 20us) op_driver.cc:348] REPLICATION: starting
1026 14:59:48.418986 (+ 86312us) log.cc:844] Serialized 960 byte log entry
1026 14:59:50.346782 (+1927796us) op_driver.cc:464] REPLICATION: finished
1026 14:59:51.452496 (+1105714us) write_op.cc:301] APPLY: starting
1026 14:59:51.459176 (+ 6680us) tablet.cc:1365] starting BulkCheckPresence
1026 14:59:51.488102 (+ 28926us) tablet.cc:1368] finished BulkCheckPresence
1026 14:59:51.537490 (+ 49388us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 14:59:51.639514 (+102024us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 14:59:51.674577 (+ 35063us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1026 14:59:51.701453 (+ 26876us) write_op.cc:312] APPLY: finished
1026 14:59:51.701694 (+ 241us) log.cc:844] Serialized 28 byte log entry
1026 14:59:52.066172 (+364478us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:52.073314 (+ 7142us) write_op.cc:454] Released schema lock
1026 14:59:52.076039 (+ 2725us) write_op.cc:341] FINISH: Updating metrics
1026 14:59:52.156575 (+ 80536us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:52.162312 (+ 5737us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1993056,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":1050975,"apply.run_cpu_time_us":36653,"apply.run_wall_time_us":672796,"num_ops":2,"prepare.queue_time_us":423981,"prepare.run_cpu_time_us":52404,"prepare.run_wall_time_us":374188,"raft.queue_time_us":1113691,"raft.run_cpu_time_us":123668,"raft.run_wall_time_us":1097984,"replication_time_us":2013884,"thread_start_us":3119490,"threads_started":5,"wal-append.queue_time_us":1197331,"wal-append.run_cpu_time_us":68387,"wal-append.run_wall_time_us":3567804}]]}
I20251026 14:59:55.632184 22697 leader_election.cc:304] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:55.640792 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 127909us
W20251026 14:59:55.641688 22624 leader_election.cc:336] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.756s (SENT)
I20251026 14:59:55.642107 22624 leader_election.cc:304] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:55.648959 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 69243.2us
I20251026 14:59:55.650687 25527 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:55.653841 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 59971.3us
I20251026 14:59:55.655050 25583 raft_consensus.cc:2804] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251026 14:59:55.655412 25527 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:55.655790 25583 raft_consensus.cc:493] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:55.656047 22622 leader_election.cc:304] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:55.656397 25583 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:55.629597 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 14:59:55.600728 25576 raft_consensus.cc:515] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:55.666047 22652 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:55.633505 25753 raft_consensus.cc:2749] T ba8784cbec044fb7b5b098961f0f94b2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251026 14:59:55.639837 25801 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:55.627988 25636 raft_consensus.cc:2749] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251026 14:59:55.675128 22659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1676) took 3097 ms. Trace:
I20251026 14:59:55.681325 22652 raft_consensus.cc:3060] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:55.678023 22659 rpcz_store.cc:276] 1026 14:59:52.577658 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:55.116455 (+2538797us) service_pool.cc:225] Handling call
1026 14:59:55.117057 (+ 602us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:55.192720 (+ 75663us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:55.192759 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:55.193572 (+ 813us) log.cc:844] Serialized 52 byte log entry
1026 14:59:55.454618 (+261046us) raft_consensus.cc:1645] Marking committed up to 2
1026 14:59:55.454660 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.454740 (+ 80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:55.674800 (+220060us) raft_consensus.cc:1685] finished
1026 14:59:55.674834 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:55.675043 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3237,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":161,"thread_start_us":260570,"threads_started":1,"wal-append.queue_time_us":388789}
I20251026 14:59:55.675863 23402 raft_consensus.cc:1240] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20251026 14:59:55.703451 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 60371.9us
W20251026 14:59:55.707484 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 65838.5us
I20251026 14:59:55.707608 22657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:55.706719 22655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32888 (request call id 160) took 1323 ms. Trace:
W20251026 14:59:55.707870 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:55.708285 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 52371.7us
I20251026 14:59:55.705430 22659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bb7c83ff5ae4d4a802850a40803bc55" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:55.710532 22803 raft_consensus.cc:2468] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:55.711144 22773 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:55.714793 25636 raft_consensus.cc:2804] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Leader election won for term 10
I20251026 14:59:55.693100 25801 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:55.718186 25801 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:55.715128 25758 consensus_queue.cc:1048] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.020s
I20251026 14:59:55.721346 25636 raft_consensus.cc:697] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:55.721829 25515 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:55.722489 25515 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:55.722667 25636 consensus_queue.cc:237] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:55.723896 22657 raft_consensus.cc:1275] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20251026 14:59:55.724261 22657 pending_rounds.cc:85] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29: Aborting all ops after (but not including) 5
I20251026 14:59:55.721166 22697 leader_election.cc:304] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:55.724550 22657 pending_rounds.cc:107] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20251026 14:59:55.725472 22730 raft_consensus.cc:2410] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251026 14:59:55.724802 22657 raft_consensus.cc:2889] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:55.725881 25758 raft_consensus.cc:2749] T 5c9a78d6a2b442eabd005c8378b68733 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251026 14:59:55.728112 22651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1699) took 1586 ms. Trace:
I20251026 14:59:55.702194 25670 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:55.728950 25753 consensus_queue.cc:1048] T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251026 14:59:55.730229 25576 leader_election.cc:290] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:55.704830 25604 raft_consensus.cc:2749] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20251026 14:59:55.730700 25670 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 10 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:55.728549 22651 rpcz_store.cc:276] 1026 14:59:54.141510 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:55.479282 (+1337772us) service_pool.cc:225] Handling call
1026 14:59:55.551796 (+ 72514us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:55.552412 (+ 616us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:55.552461 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:55.553198 (+ 737us) log.cc:844] Serialized 52 byte log entry
1026 14:59:55.661573 (+108375us) raft_consensus.cc:1645] Marking committed up to 3
1026 14:59:55.661613 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.661679 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:55.727872 (+ 66193us) raft_consensus.cc:1685] finished
1026 14:59:55.727896 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:55.728056 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3616,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":99,"thread_start_us":79062,"threads_started":1,"wal-append.queue_time_us":121692}
I20251026 14:59:55.686004 25586 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:55.735180 25586 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:55.735515 25758 consensus_queue.cc:1059] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20251026 14:59:55.737174 25732 raft_consensus.cc:3055] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 9 LEADER]: Stepping down as leader of term 9
I20251026 14:59:55.737594 25732 raft_consensus.cc:740] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:55.737778 22657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:54560 (request call id 260) took 3702 ms. Trace:
I20251026 14:59:55.738161 22657 rpcz_store.cc:276] 1026 14:59:52.035710 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:55.727445 (+3691735us) service_pool.cc:225] Handling call
1026 14:59:55.737675 (+ 10230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251026 14:59:55.738659 25732 consensus_queue.cc:260] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
W20251026 14:59:55.739310 22803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44594 (request call id 1507) took 1963 ms (client timeout 2246 ms). Trace:
W20251026 14:59:55.739768 22803 rpcz_store.cc:269] 1026 14:59:53.776071 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.058208 (+282137us) service_pool.cc:225] Handling call
1026 14:59:55.739038 (+1680830us) spinlock_profiling.cc:243] Waited 1.61 s on lock 0x7b54003336d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:55.739219 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3692610304}
I20251026 14:59:55.740510 25732 raft_consensus.cc:3060] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:55.743151 22622 leader_election.cc:304] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:55.740993 22604 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet f688f64ce4e84d419d88768435bf773f, TS 0d684445287b450a8f738b6ad4b59821 -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:55.745530 22604 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet f688f64ce4e84d419d88768435bf773f, TS 0d684445287b450a8f738b6ad4b59821 -> TS 248493ec312e4398b828c6cb00e5c4b1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:55.745635 25621 raft_consensus.cc:2749] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251026 14:59:55.746253 25702 consensus_queue.cc:1048] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20251026 14:59:55.746496 25529 consensus_queue.cc:1048] T 4129daa4fb70408193f07b8e6d25effe P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20251026 14:59:55.748181 25196 consensus_queue.cc:1059] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20251026 14:59:55.736203 22624 leader_election.cc:336] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.761s (SENT)
I20251026 14:59:55.741571 22730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35084 (request call id 1287) took 1013 ms. Trace:
I20251026 14:59:55.741721 22697 leader_election.cc:304] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:55.718701 25527 raft_consensus.cc:3055] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:55.691377 25481 consensus_queue.cc:1059] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20251026 14:59:55.759326 22624 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:55.769428 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:55.758546 22730 rpcz_store.cc:276] 1026 14:59:54.728421 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:54.734677 (+ 6256us) service_pool.cc:225] Handling call
1026 14:59:55.741316 (+1006639us) spinlock_profiling.cc:243] Waited 592 ms on lock 0x7b54000d66d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:55.741480 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1361181952}
I20251026 14:59:55.744644 22622 leader_election.cc:304] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
W20251026 14:59:55.777570 22623 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:55.780306 22652 raft_consensus.cc:2468] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:55.708233 22655 rpcz_store.cc:276] 1026 14:59:54.383396 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:55.596858 (+1213462us) service_pool.cc:225] Handling call
1026 14:59:55.607059 (+ 10201us) raft_consensus.cc:1471] Updating replica for 0 ops
1026 14:59:55.653420 (+ 46361us) raft_consensus.cc:1534] Early marking committed up to index 8
1026 14:59:55.653800 (+ 380us) log.cc:844] Serialized 14 byte log entry
1026 14:59:55.703817 (+ 50017us) raft_consensus.cc:1539] Triggering prepare for 0 ops
1026 14:59:55.703876 (+ 59us) raft_consensus.cc:1645] Marking committed up to 8
1026 14:59:55.706516 (+ 2640us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.706581 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:55.706673 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14593,"prepare.queue_time_us":930,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":23680,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":995,"thread_start_us":26823,"threads_started":1}
I20251026 14:59:55.730640 25753 consensus_queue.cc:1048] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
W20251026 14:59:55.790501 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 54348.5us
I20251026 14:59:55.793365 25801 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:55.793388 25515 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:55.768739 25527 raft_consensus.cc:740] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
W20251026 14:59:55.787218 23367 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 76817.4us
I20251026 14:59:55.778064 25522 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:55.796806 25604 raft_consensus.cc:2749] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251026 14:59:55.797489 25576 raft_consensus.cc:2749] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:55.797315 25522 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:55.784705 25586 raft_consensus.cc:2804] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:55.804194 25602 consensus_queue.cc:1059] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20251026 14:59:55.807956 25586 raft_consensus.cc:697] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:55.808826 25602 raft_consensus.cc:3055] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:55.809303 25602 raft_consensus.cc:740] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
I20251026 14:59:55.809077 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:55.810601 22801 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 7.
I20251026 14:59:55.783166 25193 consensus_queue.cc:1059] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20251026 14:59:55.809053 25586 consensus_queue.cc:237] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:55.816059 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:55.818351 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 73856.5us
I20251026 14:59:55.819978 25534 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:55.804284 25527 consensus_queue.cc:260] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:55.820739 25534 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:55.821395 25522 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:55.822582 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:55.823025 25534 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:55.823329 23402 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:55.819983 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:55.826817 25602 consensus_queue.cc:260] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:55.828577 25602 raft_consensus.cc:3060] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:55.833191 25819 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:55.835497 25819 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:55.841382 25670 consensus_queue.cc:1059] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20251026 14:59:55.841838 22622 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:55.842444 25604 consensus_queue.cc:1048] T 5c9a78d6a2b442eabd005c8378b68733 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 14:59:55.856045 25824 raft_consensus.cc:2804] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:55.843148 25576 consensus_queue.cc:1059] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 14:59:55.856479 25824 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:55.856858 25824 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:55.857031 25576 raft_consensus.cc:3055] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Stepping down as leader of term 8
I20251026 14:59:55.857506 25576 raft_consensus.cc:740] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:59:55.857594 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 115952us
I20251026 14:59:55.858530 25576 consensus_queue.cc:260] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:55.859021 25481 consensus_queue.cc:1059] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20251026 14:59:55.855139 25583 raft_consensus.cc:515] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:55.860491 25753 raft_consensus.cc:3055] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 8 LEADER]: Stepping down as leader of term 8
W20251026 14:59:55.843467 25831 log.cc:927] Time spent T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.053s user 0.000s sys 0.003s
I20251026 14:59:55.786474 25783 raft_consensus.cc:515] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:55.863234 25783 leader_election.cc:290] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Requested vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:55.863683 22622 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:55.864656 25604 raft_consensus.cc:2749] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20251026 14:59:55.866608 22622 leader_election.cc:304] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:55.845849 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2b59eefda3b4f2d93934f02c4981c62" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:55.837105 25241 raft_consensus.cc:3055] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:55.873584 25241 raft_consensus.cc:740] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:55.859907 25576 raft_consensus.cc:3060] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:55.874492 22656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1685) took 2118 ms. Trace:
I20251026 14:59:55.874711 25241 consensus_queue.cc:260] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:55.874969 22656 rpcz_store.cc:276] 1026 14:59:53.756113 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:55.285631 (+1529518us) service_pool.cc:225] Handling call
1026 14:59:55.317434 (+ 31803us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:55.568536 (+251102us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:55.568566 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:55.581983 (+ 13417us) log.cc:844] Serialized 26 byte log entry
1026 14:59:55.692805 (+110822us) raft_consensus.cc:1645] Marking committed up to 6
1026 14:59:55.692863 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.693137 (+ 274us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:55.874237 (+181100us) raft_consensus.cc:1685] finished
1026 14:59:55.874257 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:55.874424 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9384,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":102,"thread_start_us":314099,"threads_started":2,"wal-append.queue_time_us":188740}
I20251026 14:59:55.873167 22655 raft_consensus.cc:2468] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 7.
I20251026 14:59:55.877626 25527 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:55.876118 25241 raft_consensus.cc:3060] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
W20251026 14:59:55.878049 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:55.850773 25534 consensus_queue.cc:1059] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20251026 14:59:55.882474 25534 raft_consensus.cc:3055] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 8 LEADER]: Stepping down as leader of term 8
I20251026 14:59:55.883080 25534 raft_consensus.cc:740] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
W20251026 14:59:55.867604 25825 log.cc:927] Time spent T 4dea3b45142b41baacfa105498c9e04e P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20251026 14:59:55.869673 22650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1697) took 1796 ms. Trace:
I20251026 14:59:55.884491 25534 consensus_queue.cc:260] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:55.885565 22650 rpcz_store.cc:276] 1026 14:59:54.073182 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:55.490200 (+1417018us) service_pool.cc:225] Handling call
1026 14:59:55.554291 (+ 64091us) raft_consensus.cc:1471] Updating replica for 1 ops
1026 14:59:55.554766 (+ 475us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:55.554795 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1026 14:59:55.555326 (+ 531us) log.cc:844] Serialized 26 byte log entry
1026 14:59:55.589091 (+ 33765us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:55.613152 (+ 24061us) log.cc:844] Serialized 14 byte log entry
1026 14:59:55.616743 (+ 3591us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.616829 (+ 86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:55.647741 (+ 30912us) spinlock_profiling.cc:243] Waited 6.44 ms on lock 0x7b48000801d8. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d606b971 00007fd8d6062a37 00007fd8d60628ca 00007fd8d6159a0f 00007fd8d61596d7 00007fd8d616dcfc 00007fd8d616dca8 00007fd8d616dc28 00007fd8d616dbe3 00007fd8d616ce8c 00007fd8d50aa326
1026 14:59:55.852016 (+204275us) spinlock_profiling.cc:243] Waited 6.75 ms on lock 0x7b18001be1a0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce0b1992 00007fd8ce0b3d61 00007fd8ce0b3d19 00007fd8ce0b3ca9 00007fd8ce0b3c71 00007fd8ce0b2f6d 0000000000379254 0000000000379099 00007fd8ce094d1a
1026 14:59:55.869414 (+ 17398us) raft_consensus.cc:1685] finished
1026 14:59:55.869441 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:55.869608 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":48976,"prepare.queue_time_us":24743,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":4835,"raft.run_cpu_time_us":3254,"raft.run_wall_time_us":30111,"spinlock_wait_cycles":30342272,"thread_start_us":33360,"threads_started":1,"wal-append.queue_time_us":72085}
W20251026 14:59:55.844879 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:55.889178 22658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1694) took 1958 ms. Trace:
I20251026 14:59:55.860977 25753 raft_consensus.cc:740] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:55.890347 25753 consensus_queue.cc:260] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:55.889693 22658 rpcz_store.cc:276] 1026 14:59:53.930853 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:55.404147 (+1473294us) service_pool.cc:225] Handling call
1026 14:59:55.404592 (+ 445us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:55.438189 (+ 33597us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:55.438226 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:55.438935 (+ 709us) log.cc:844] Serialized 52 byte log entry
1026 14:59:55.627537 (+188602us) raft_consensus.cc:1645] Marking committed up to 4
1026 14:59:55.634975 (+ 7438us) log.cc:844] Serialized 14 byte log entry
1026 14:59:55.635706 (+ 731us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.697147 (+ 61441us) spinlock_profiling.cc:243] Waited 1.69 ms on lock 0x7b18002d2da0. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce07f986 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9
1026 14:59:55.697772 (+ 625us) spinlock_profiling.cc:243] Waited 33.1 ms on lock 0x7b54002a85d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d115e9a1 00007fd8d115d3de 00007fd8d61059d9 00007fd8cf69fd34 00007fd8cf69fcc2 00007fd8cf69fbf1 00007fd8cf69fb6c 00007fd8cf69ee22 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:55.697833 (+ 61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:55.888928 (+191095us) raft_consensus.cc:1685] finished
1026 14:59:55.888952 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:55.889120 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30577,"prepare.queue_time_us":164,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":13277,"raft.run_cpu_time_us":354,"raft.run_wall_time_us":54199,"spinlock_wait_cycles":80007168,"thread_start_us":188142,"threads_started":1,"wal-append.queue_time_us":247445}
W20251026 14:59:55.889391 22624 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:55.893580 22624 leader_election.cc:336] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.634s (SENT)
I20251026 14:59:55.894433 22624 leader_election.cc:304] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:55.872471 25801 raft_consensus.cc:2749] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251026 14:59:55.895685 25702 raft_consensus.cc:3055] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:55.895731 25804 raft_consensus.cc:3055] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:55.896478 25702 raft_consensus.cc:740] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:55.896564 25804 raft_consensus.cc:740] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:55.898131 25804 consensus_queue.cc:260] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:55.898204 25702 consensus_queue.cc:260] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:55.899775 25804 raft_consensus.cc:3060] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:55.900100 25702 raft_consensus.cc:3060] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:55.871728 25824 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:55.906805 25753 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:55.907059 25824 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:55.909600 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 50926.6us
I20251026 14:59:55.911058 25529 consensus_queue.cc:1059] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20251026 14:59:55.911154 22697 leader_election.cc:304] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters: df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:55.912325 25709 raft_consensus.cc:3055] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Stepping down as leader of term 6
I20251026 14:59:55.912750 25709 raft_consensus.cc:740] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:55.913622 25709 consensus_queue.cc:260] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:55.914108 25604 raft_consensus.cc:2804] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251026 14:59:55.914620 25604 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:55.915611 22650 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:55.917235 25709 raft_consensus.cc:3060] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:55.914978 25604 raft_consensus.cc:3060] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Advancing to term 10
W20251026 14:59:55.922816 22624 leader_election.cc:336] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.724s (SENT)
W20251026 14:59:55.923872 22624 leader_election.cc:336] T 5b0536aab3b541a2a846d187e111c09c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.796s (SENT)
W20251026 14:59:55.924786 22624 leader_election.cc:336] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.801s (SENT)
I20251026 14:59:55.925457 25819 raft_consensus.cc:2749] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20251026 14:59:55.913968 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:55.926388 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fbb846c2db3b4f44b2245a7493b36c7c" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 11 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:55.927066 25583 leader_election.cc:290] T 11ae57d90c614ce2993629f95b777120 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:55.930914 22650 raft_consensus.cc:1275] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20251026 14:59:55.932597 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 66128.8us
I20251026 14:59:55.925191 22624 leader_election.cc:304] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:55.921168 25534 raft_consensus.cc:3060] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
W20251026 14:59:55.947309 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:55.938051 25819 raft_consensus.cc:2749] T aceef9b41bab4ad3bf67b4d597a7fe11 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:55.969295 22622 leader_election.cc:304] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:55.950856 25783 raft_consensus.cc:2804] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election won for term 9
W20251026 14:59:55.974423 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:55.951103 25801 consensus_queue.cc:1059] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20251026 14:59:55.974097 25783 raft_consensus.cc:493] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:55.995683 25523 raft_consensus.cc:2804] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251026 14:59:55.971868 25824 consensus_queue.cc:1059] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
W20251026 14:59:55.998332 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 73591.3us
I20251026 14:59:55.978878 25529 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:56.001636 22653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54574 (request call id 1700) took 1821 ms. Trace:
I20251026 14:59:56.003378 22622 leader_election.cc:400] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 election: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 9. Current term is 10.
I20251026 14:59:56.005342 22622 leader_election.cc:403] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20251026 14:59:56.001252 25783 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:56.000517 25523 raft_consensus.cc:493] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:56.010921 25523 raft_consensus.cc:3060] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Advancing to term 10
W20251026 14:59:56.007580 22553 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38880 (request call id 173) took 30551 ms (client timeout 15000 ms). Trace:
W20251026 14:59:56.006043 22624 leader_election.cc:336] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.834s (SENT)
I20251026 14:59:56.008345 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:56.018428 22624 leader_election.cc:304] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:56.004251 25583 raft_consensus.cc:493] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:56.019654 25583 raft_consensus.cc:515] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:56.005836 22552 catalog_manager.cc:5649] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } }
I20251026 14:59:56.022926 25196 raft_consensus.cc:493] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:56.022764 22552 catalog_manager.cc:5649] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 2 to 7, leader changed from df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 7 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:56.023478 25196 raft_consensus.cc:515] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:56.004071 25801 consensus_queue.cc:1243] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer 0d684445287b450a8f738b6ad4b59821 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20251026 14:59:56.029603 22799 raft_consensus.cc:1240] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 11 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 10. Current term is 11. Ops: []
I20251026 14:59:56.029628 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4129daa4fb70408193f07b8e6d25effe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:56.031168 22803 raft_consensus.cc:1275] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20251026 14:59:56.030499 25758 raft_consensus.cc:3055] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Stepping down as leader of term 4
I20251026 14:59:56.031816 25758 raft_consensus.cc:740] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:56.024670 22552 catalog_manager.cc:5649] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 4 to 6, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 6 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:56.002681 22653 rpcz_store.cc:276] 1026 14:59:54.180370 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:55.492627 (+1312257us) service_pool.cc:225] Handling call
1026 14:59:55.534853 (+ 42226us) raft_consensus.cc:1471] Updating replica for 2 ops
1026 14:59:55.610918 (+ 76065us) raft_consensus.cc:1534] Early marking committed up to index 0
1026 14:59:55.610948 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
1026 14:59:55.616187 (+ 5239us) log.cc:844] Serialized 52 byte log entry
1026 14:59:55.715897 (+ 99710us) spinlock_profiling.cc:243] Waited 3.83 ms on lock 0x7b1800099460. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d6ef49a1 00007fd8d6f01371 00007fd8ce0942f7 00007fd8d6e6d219 00007fd8ce0b0c4b 00007fd8ce0aeeef 00007fd8ce0b11b2 00007fd8d0cf82bb 00007fd8d0cfddc3 00007fd8d0cfe171 00007fd8d111965f
1026 14:59:55.798761 (+ 82864us) raft_consensus.cc:1645] Marking committed up to 5
1026 14:59:55.798811 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
1026 14:59:55.798882 (+ 71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1026 14:59:55.998192 (+199310us) raft_consensus.cc:1685] finished
1026 14:59:55.998205 (+ 13us) raft_consensus.cc:1691] UpdateReplicas() finished
1026 14:59:56.001567 (+ 3362us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7857,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":21312,"spinlock_wait_cycles":8808192,"thread_start_us":82677,"threads_started":1,"wal-append.queue_time_us":324216}
I20251026 14:59:56.041718 22803 raft_consensus.cc:1275] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Refusing update from remote peer 562bb1e1c8e445b8baecd29c3669658e: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20251026 14:59:56.042526 22622 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:56.045619 25196 leader_election.cc:290] T 988275d9b2574d0381d8f21ee8d99381 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:56.046931 23400 raft_consensus.cc:1240] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 10 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20251026 14:59:56.048143 25828 consensus_queue.cc:1059] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251026 14:59:56.046955 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:56.049256 25828 raft_consensus.cc:3055] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 14:59:56.049674 25828 raft_consensus.cc:740] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:56.049918 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "469842eacb94455aaff74df534ff0bf1" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:56.050514 25828 consensus_queue.cc:260] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:56.052002 25828 raft_consensus.cc:3060] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 2 FOLLOWER]: Advancing to term 3
I20251026 14:59:56.039165 25602 raft_consensus.cc:2749] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 9. Current term is 10.
I20251026 14:59:56.050565 22801 raft_consensus.cc:2468] T 469842eacb94455aaff74df534ff0bf1 P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 6.
I20251026 14:59:56.056643 25523 raft_consensus.cc:515] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:56.057212 25576 raft_consensus.cc:3055] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Stepping down as leader of term 8
I20251026 14:59:56.050920 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:56.065949 25576 raft_consensus.cc:740] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:59:56.066722 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 62628us
I20251026 14:59:56.066951 25576 consensus_queue.cc:260] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:56.037477 25758 consensus_queue.cc:260] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:56.068255 25576 raft_consensus.cc:3060] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:56.068849 25758 raft_consensus.cc:3060] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Advancing to term 5
I20251026 14:59:56.003043 25529 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:56.071259 25529 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:56.072533 25583 leader_election.cc:290] T 21e9405b016a4df592291cba0e961b07 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:56.067420 22624 leader_election.cc:336] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.926s (SENT)
W20251026 14:59:56.016953 22553 rpcz_store.cc:269] 1026 14:59:25.442624 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:25.443027 (+ 403us) service_pool.cc:225] Handling call
1026 14:59:26.716209 (+1273182us) spinlock_profiling.cc:243] Waited 804 ms on lock 0x7b54003af554. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c7fe0 00007fd8d1166aca 00007fd8d6ed7bbc 00007fd8d6fd1e7e 00007fd8d54d24a4 00007fd8d54d2432 00007fd8d54d2361 00007fd8d54d22dc 00007fd8d54d1592 00007fd8cf49decc 00007fd8cf49d306
1026 14:59:55.978628 (+29262419us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 14:59:45.034168 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1026 14:59:45.035066 (+ 898us) write_op.cc:432] Acquiring schema lock in shared mode
1026 14:59:45.035161 (+ 95us) write_op.cc:435] Acquired schema lock
1026 14:59:45.035206 (+ 45us) tablet.cc:660] Decoding operations
1026 14:59:45.276621 (+241415us) write_op.cc:620] Acquiring the partition lock for write op
1026 14:59:45.295501 (+ 18880us) write_op.cc:641] Partition lock acquired for write op
1026 14:59:45.298575 (+ 3074us) tablet.cc:683] Acquiring locks for 25 operations
1026 14:59:45.307756 (+ 9181us) tablet.cc:699] Row locks acquired
1026 14:59:45.307793 (+ 37us) write_op.cc:260] PREPARE: finished
1026 14:59:45.308017 (+ 224us) write_op.cc:270] Start()
1026 14:59:45.308200 (+ 183us) write_op.cc:276] Timestamp: P: 1761490785307940 usec, L: 0
1026 14:59:45.308221 (+ 21us) op_driver.cc:348] REPLICATION: starting
1026 14:59:45.356369 (+ 48148us) log.cc:844] Serialized 9624 byte log entry
1026 14:59:45.673818 (+317449us) op_driver.cc:464] REPLICATION: finished
1026 14:59:45.865593 (+191775us) write_op.cc:301] APPLY: starting
1026 14:59:45.865726 (+ 133us) tablet.cc:1365] starting BulkCheckPresence
1026 14:59:45.939032 (+ 73306us) tablet.cc:1368] finished BulkCheckPresence
1026 14:59:45.939227 (+ 195us) tablet.cc:1370] starting ApplyRowOperation cycle
1026 14:59:46.385681 (+446454us) tablet.cc:1381] finished ApplyRowOperation cycle
1026 14:59:46.517791 (+132110us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=25
1026 14:59:46.517886 (+ 95us) write_op.cc:312] APPLY: finished
1026 14:59:46.518462 (+ 576us) log.cc:844] Serialized 169 byte log entry
1026 14:59:46.518687 (+ 225us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:46.518908 (+ 221us) write_op.cc:454] Released schema lock
1026 14:59:46.518993 (+ 85us) write_op.cc:341] FINISH: Updating metrics
1026 14:59:46.519539 (+ 546us) write_op.cc:489] Releasing partition, row and schema locks
1026 14:59:46.519570 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2418330,"reported_tablets":45,"spinlock_wait_cycles":1850096000,"child_traces":[["op",{"apply.queue_time_us":174401,"apply.run_cpu_time_us":10708,"apply.run_wall_time_us":661723,"num_ops":25,"prepare.queue_time_us":213021,"prepare.run_cpu_time_us":25385,"prepare.run_wall_time_us":397011,"raft.queue_time_us":118103,"raft.run_cpu_time_us":13486,"raft.run_wall_time_us":164095,"replication_time_us":365393,"thread_start_us":302554,"threads_started":4,"wal-append.queue_time_us":111447,"wal-append.run_cpu_time_us":18544,"wal-append.run_wall_time_us":2094010}]]}
I20251026 14:59:56.074653 22624 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:56.061379 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 88902us
I20251026 14:59:56.061519 25604 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:56.078804 22552 catalog_manager.cc:5649] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 2 to 7, leader changed from df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 7 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:56.082482 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ea1e7da57b65433b9e01cad8c4adeb75" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:59:56.087608 22774 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:56.089303 22697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:56.089730 22697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:56.043115 25824 raft_consensus.cc:2749] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:56.098456 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:56.099051 22802 raft_consensus.cc:3060] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:56.088539 25602 raft_consensus.cc:2804] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:56.098310 22552 catalog_manager.cc:5649] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 4 to 7, leader changed from df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 7 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
W20251026 14:59:56.101827 22700 connection.cc:232] RPC call timeout handler was delayed by -2.642077793999988s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3537s for a response
I20251026 14:59:56.101518 25801 raft_consensus.cc:493] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:56.105134 25801 raft_consensus.cc:515] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:56.105163 25732 raft_consensus.cc:493] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 248493ec312e4398b828c6cb00e5c4b1)
I20251026 14:59:56.106882 25732 raft_consensus.cc:515] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:56.107635 25801 leader_election.cc:290] T 5373ef47363d446fbf9d697ed507e754 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
W20251026 14:59:56.104499 22700 connection.cc:232] RPC call timeout handler was delayed by -2.018280265000044s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2947s for a response
W20251026 14:59:56.108929 22700 connection.cc:232] RPC call timeout handler was delayed by -1.7794554260000268s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1549s for a response
W20251026 14:59:56.109306 22700 connection.cc:232] RPC call timeout handler was delayed by -1.7668554260000064s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1563s for a response
I20251026 14:59:56.093654 25586 raft_consensus.cc:2749] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251026 14:59:56.109946 22700 connection.cc:232] RPC call timeout handler was delayed by -1.74435542599997s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15880000000000002s for a response
W20251026 14:59:56.110261 22700 connection.cc:232] RPC call timeout handler was delayed by -1.281180264999989s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.37660000000000005s for a response
I20251026 14:59:56.101526 25602 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:56.112496 25602 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:56.118402 25604 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:56.119411 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:56.122098 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:56.122781 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:56.123450 22650 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 6.
I20251026 14:59:56.123513 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:56.124219 22654 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:56.126367 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:56.127048 22732 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 7.
I20251026 14:59:56.129482 25604 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:56.130115 25604 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:56.131688 22654 raft_consensus.cc:2468] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 10.
W20251026 14:59:56.139508 22700 connection.cc:232] RPC call timeout handler was delayed by -1.203455426000005s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2189s for a response
I20251026 14:59:56.093880 25783 raft_consensus.cc:515] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:56.142408 25732 leader_election.cc:290] T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:56.144399 25523 leader_election.cc:290] T 3ea63385804c40689502aefb2b695e78 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:56.144471 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:56.152909 23370 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:56.154351 25527 raft_consensus.cc:2749] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:56.155056 22622 leader_election.cc:400] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 10. Current term is 11.
I20251026 14:59:56.155484 22622 leader_election.cc:403] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20251026 14:59:56.155928 23370 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:56.156206 25819 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Advancing to term 11
I20251026 14:59:56.156926 25527 raft_consensus.cc:2804] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:56.157346 25527 raft_consensus.cc:493] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:56.157694 25527 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:56.142551 25783 leader_election.cc:290] T 2a4022b371fc46c89fbc9a5470cda03e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:56.162686 23370 leader_election.cc:304] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:56.164739 25828 raft_consensus.cc:2804] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 10 FOLLOWER]: Leader election won for term 10
I20251026 14:59:56.165443 25828 raft_consensus.cc:697] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [term 10 LEADER]: Becoming Leader. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:56.166085 23402 raft_consensus.cc:2468] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 3.
I20251026 14:59:56.166635 25828 consensus_queue.cc:237] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:56.162977 25833 log.cc:927] Time spent T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20251026 14:59:56.172324 22802 raft_consensus.cc:2468] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 10.
I20251026 14:59:56.171617 22552 catalog_manager.cc:5649] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 4 to 7, leader changed from df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 7 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:56.186662 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "524d8339d5244ca98e86f0ff2fd02b67" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:56.184096 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:56.188063 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
W20251026 14:59:56.191362 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:56.192317 22800 raft_consensus.cc:2410] T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562bb1e1c8e445b8baecd29c3669658e for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20251026 14:59:56.193975 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:56.194772 22804 raft_consensus.cc:3060] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
W20251026 14:59:56.194343 22624 leader_election.cc:336] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 2.058s (SENT)
I20251026 14:59:56.186887 22552 catalog_manager.cc:5649] T 3103efcd5601429a95eeb7a270f6fab1 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 4 to 9, leader changed from df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 9 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:56.203509 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "129989aafdc84d6b8e56fe97c7182fff" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:56.213850 25840 consensus_queue.cc:1059] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20251026 14:59:56.215152 22700 leader_election.cc:304] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:56.215271 25840 raft_consensus.cc:3055] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 9 LEADER]: Stepping down as leader of term 9
I20251026 14:59:56.215828 25840 raft_consensus.cc:740] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:56.216449 22700 leader_election.cc:304] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:56.217514 25753 raft_consensus.cc:2749] T 4e49f6058b03447ca17f7a94a8b071d8 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20251026 14:59:56.217772 25783 raft_consensus.cc:2804] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 11 FOLLOWER]: Leader pre-election won for term 12
W20251026 14:59:56.218240 22624 leader_election.cc:336] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 2.073s (SENT)
I20251026 14:59:56.218313 25783 raft_consensus.cc:493] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:56.222935 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "627ebd70757a47b18827c38f071b5261" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 12 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:56.226662 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4dea3b45142b41baacfa105498c9e04e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:56.180682 25602 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:56.221415 22552 catalog_manager.cc:5649] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 4 to 5, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 5 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:56.233731 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:56.233834 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ae57d90c614ce2993629f95b777120" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:56.234345 22800 raft_consensus.cc:2393] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 7: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
W20251026 14:59:56.236881 22624 leader_election.cc:336] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 2.073s (SENT)
I20251026 14:59:56.237243 22624 leader_election.cc:304] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:56.239225 22807 raft_consensus.cc:2393] T 11ae57d90c614ce2993629f95b777120 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 10: Already voted for candidate 562bb1e1c8e445b8baecd29c3669658e in this term.
I20251026 14:59:56.239395 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "45fb267b6944430e8237d9baad843789" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:56.240442 22697 leader_election.cc:304] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e; no voters:
I20251026 14:59:56.241128 22697 leader_election.cc:304] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1
I20251026 14:59:56.220650 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aceef9b41bab4ad3bf67b4d597a7fe11" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:56.241892 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:56.242695 22806 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
I20251026 14:59:56.223111 25783 raft_consensus.cc:3060] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 11 FOLLOWER]: Advancing to term 12
I20251026 14:59:56.246640 22700 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:56.248071 22552 catalog_manager.cc:5649] T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 4 to 5, leader changed from df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 5 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:56.243465 25801 raft_consensus.cc:2749] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 14:59:56.242038 25753 consensus_queue.cc:1048] T 7f59b567deee4fd6bc1428954f8af0ad P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.012s
I20251026 14:59:56.253084 25604 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:56.241961 25523 raft_consensus.cc:2749] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251026 14:59:56.241915 25732 consensus_queue.cc:1048] T 6b172938cfc843318b60a7fc78190d2c P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20251026 14:59:56.243649 25241 raft_consensus.cc:2804] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:56.254750 25241 raft_consensus.cc:493] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:56.255273 25241 raft_consensus.cc:3060] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:56.231431 25583 consensus_queue.cc:1059] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20251026 14:59:56.199815 25755 consensus_queue.cc:1048] T 1badbeef1a1c4427bbfa1b762c7d0739 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.038s
I20251026 14:59:56.256049 22552 catalog_manager.cc:5649] T 7f59b567deee4fd6bc1428954f8af0ad P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 1 to 4, leader changed from df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 4 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:56.262893 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2a4022b371fc46c89fbc9a5470cda03e" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:56.265197 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5b0536aab3b541a2a846d187e111c09c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:56.266594 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9edd43dec704b6e8351cebe3921d663" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:56.267396 22802 raft_consensus.cc:3060] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:56.260733 22552 catalog_manager.cc:5649] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 3 to 7, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 7 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:56.261852 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
W20251026 14:59:56.269858 22624 leader_election.cc:336] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 2.099s (SENT)
I20251026 14:59:56.270545 22624 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:56.270810 22697 leader_election.cc:304] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:56.248078 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:56.279453 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea63385804c40689502aefb2b695e78" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:56.275233 22552 catalog_manager.cc:5649] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
W20251026 14:59:56.283470 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:56.286592 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e7ddd2458b64111839b7740daabf522" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:56.216979 25840 consensus_queue.cc:260] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:56.294301 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:56.295189 22650 raft_consensus.cc:3060] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:56.295750 22656 raft_consensus.cc:1275] T 3b18a8235bfa4d2cb35c7976ce46e14c P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20251026 14:59:56.297837 22653 raft_consensus.cc:3060] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:56.297405 22552 catalog_manager.cc:5649] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 4 to 7, leader changed from df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 7 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:56.298977 22658 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37e63cca2eb94ba8ac9b6c8cdf99c4a5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:56.300886 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5618ccace02e47938463dce9ca26e300" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:56.307711 22654 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2979554f0ac542ab8cfcb32c7d2778d5" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:56.279423 25523 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:56.312121 22656 raft_consensus.cc:1240] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Rejecting Update request from peer 0d684445287b450a8f738b6ad4b59821 for earlier term 7. Current term is 8. Ops: []
I20251026 14:59:56.313616 22654 raft_consensus.cc:2468] T 2979554f0ac542ab8cfcb32c7d2778d5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 9.
I20251026 14:59:56.309253 22552 catalog_manager.cc:5649] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 4 to 5, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 5 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:56.255334 25527 raft_consensus.cc:515] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } attrs { replace: true } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:56.289891 25753 raft_consensus.cc:2749] T 143c213a935041868ad3b0d9584ad863 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251026 14:59:56.317514 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "21e9405b016a4df592291cba0e961b07" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:56.320313 22774 leader_election.cc:304] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:56.326231 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 90127us
I20251026 14:59:56.327491 22650 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 9.
W20251026 14:59:56.328599 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.4:43341, user_credentials={real_user=slave}} blocked reactor thread for 79434.2us
I20251026 14:59:56.330655 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:56.330514 22552 catalog_manager.cc:5649] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 5 to 6, leader changed from df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 6 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:56.331481 22650 raft_consensus.cc:2393] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 7: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
I20251026 14:59:56.332244 22552 catalog_manager.cc:5649] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 3 to 6, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 6 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:56.323572 25702 consensus_queue.cc:1048] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.010s
W20251026 14:59:56.334973 22699 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:56.336869 22552 catalog_manager.cc:5649] T 9423e278399742f48ca9ee203d7f92cb P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 6 to 7, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 7 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:56.343894 22700 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
W20251026 14:59:56.347383 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 77593.5us
I20251026 14:59:56.341368 22552 catalog_manager.cc:5649] T ba8784cbec044fb7b5b098961f0f94b2 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 2 to 5, leader changed from df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 5 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:56.349365 22552 catalog_manager.cc:5649] T 5c9a78d6a2b442eabd005c8378b68733 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 3 to 7, leader changed from 0d684445287b450a8f738b6ad4b59821 (127.21.2.3) to 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2). New cstate: current_term: 7 leader_uuid: "562bb1e1c8e445b8baecd29c3669658e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:56.326390 25801 raft_consensus.cc:2804] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 3 FOLLOWER]: Leader election won for term 3
I20251026 14:59:56.339341 25702 consensus_queue.cc:1243] T 45fb267b6944430e8237d9baad843789 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Peer df3a0d1619ee48e8ab68ee0543a72a29 log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20251026 14:59:56.356928 22624 leader_election.cc:336] T 2a4022b371fc46c89fbc9a5470cda03e P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 2.102s (SENT)
I20251026 14:59:56.358343 25801 raft_consensus.cc:697] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [term 3 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:56.358877 22650 raft_consensus.cc:1275] T e2b59eefda3b4f2d93934f02c4981c62 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20251026 14:59:56.359414 25801 consensus_queue.cc:237] T 02799da19b36403499a63101b8418ffe P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:56.365327 22804 raft_consensus.cc:2468] T 2a4022b371fc46c89fbc9a5470cda03e P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 562bb1e1c8e445b8baecd29c3669658e in term 9.
I20251026 14:59:56.370985 25755 consensus_queue.cc:1059] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 14:59:56.371819 22700 leader_election.cc:304] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:56.372938 25481 raft_consensus.cc:3055] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Stepping down as leader of term 7
I20251026 14:59:56.362339 25602 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:56.373571 25583 raft_consensus.cc:2749] T 45fb267b6944430e8237d9baad843789 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20251026 14:59:56.373361 25481 raft_consensus.cc:740] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:56.376011 25481 consensus_queue.cc:260] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:56.377511 25481 raft_consensus.cc:3060] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:56.390661 25527 leader_election.cc:290] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:56.352011 25529 raft_consensus.cc:2804] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20251026 14:59:56.394524 25529 raft_consensus.cc:493] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
W20251026 14:59:56.373335 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:56.395159 25529 raft_consensus.cc:3060] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Advancing to term 9
I20251026 14:59:56.362470 25840 raft_consensus.cc:3060] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:56.399679 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5373ef47363d446fbf9d697ed507e754" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
W20251026 14:59:56.400650 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 43746.8us
W20251026 14:59:56.401440 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:56.404449 22655 raft_consensus.cc:1275] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 4. (term mismatch)
I20251026 14:59:56.404778 22655 pending_rounds.cc:85] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29: Aborting all ops after (but not including) 3
I20251026 14:59:56.404958 22655 pending_rounds.cc:107] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20251026 14:59:56.405133 22655 raft_consensus.cc:2889] T f688f64ce4e84d419d88768435bf773f P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251026 14:59:56.406584 22624 leader_election.cc:336] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 2.140s (SENT)
I20251026 14:59:56.406685 22802 raft_consensus.cc:2468] T c9edd43dec704b6e8351cebe3921d663 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
I20251026 14:59:56.407114 22624 leader_election.cc:304] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:56.411660 22653 raft_consensus.cc:1275] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20251026 14:59:56.363122 25753 raft_consensus.cc:2804] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:56.414417 25602 raft_consensus.cc:2749] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:56.414803 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7f59b567deee4fd6bc1428954f8af0ad" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:56.412500 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:56.419643 22799 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:56.403672 25819 raft_consensus.cc:2749] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645) with higher term. Message: Invalid argument: T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 10. Current term is 11.
I20251026 14:59:56.425197 25753 raft_consensus.cc:697] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [term 8 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:56.429584 25753 consensus_queue.cc:237] T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:56.440423 22650 raft_consensus.cc:2410] T 7f59b567deee4fd6bc1428954f8af0ad P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20251026 14:59:56.441802 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:56.465507 22774 leader_election.cc:304] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:56.466531 25196 raft_consensus.cc:2804] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader election won for term 10
W20251026 14:59:56.467315 22624 leader_election.cc:336] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.926s (SENT)
I20251026 14:59:56.468380 22624 leader_election.cc:304] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:56.472167 22799 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 8.
I20251026 14:59:56.474946 22655 raft_consensus.cc:1275] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Refusing update from remote peer 0d684445287b450a8f738b6ad4b59821: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20251026 14:59:56.491989 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82201a13e5354c8ea3b41436df180125" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:56.494542 25844 log.cc:927] Time spent T aceef9b41bab4ad3bf67b4d597a7fe11 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 0.164s user 0.000s sys 0.001s
I20251026 14:59:56.496558 23402 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
I20251026 14:59:56.473511 25523 raft_consensus.cc:2749] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:56.477334 25196 raft_consensus.cc:697] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [term 10 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:56.498426 25196 consensus_queue.cc:237] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:56.503103 25636 raft_consensus.cc:2749] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:56.473409 25241 raft_consensus.cc:515] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:56.504243 22653 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
W20251026 14:59:56.506224 25846 log.cc:927] Time spent T 21e9405b016a4df592291cba0e961b07 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20251026 14:59:56.511130 25241 leader_election.cc:290] T 469842eacb94455aaff74df534ff0bf1 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:56.519663 25783 raft_consensus.cc:515] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:56.537025 25783 leader_election.cc:290] T 627ebd70757a47b18827c38f071b5261 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 12 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:57.385828 21512 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20251026 14:59:57.542557 22587 debug-util.cc:398] Leaking SignalData structure 0x7b080033d420 after lost signal to thread 21515
I20251026 14:59:57.734088 25852 raft_consensus.cc:493] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:58.247732 22658 consensus_meta.cc:287] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29: Time spent flushing consensus metadata: real 1.809s user 0.011s sys 0.008s
I20251026 14:59:58.256146 25481 raft_consensus.cc:493] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:58.632841 25852 raft_consensus.cc:515] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:58.633378 22587 debug-util.cc:398] Leaking SignalData structure 0x7b0800330b00 after lost signal to thread 22590
I20251026 14:59:58.633090 25481 raft_consensus.cc:515] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
W20251026 14:59:58.633790 22587 debug-util.cc:398] Leaking SignalData structure 0x7b080003a740 after lost signal to thread 22604
I20251026 14:59:58.634967 25852 leader_election.cc:290] T 3b18a8235bfa4d2cb35c7976ce46e14c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:58.635000 22587 debug-util.cc:398] Leaking SignalData structure 0x7b08001cd7c0 after lost signal to thread 22683
W20251026 14:59:58.321321 22700 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.949s user 0.017s sys 0.019s
I20251026 14:59:58.632830 22658 raft_consensus.cc:2468] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 11.
W20251026 14:59:58.635903 22700 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.263s user 0.018s sys 0.019s
W20251026 14:59:58.636498 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 2.26457e+06us
I20251026 14:59:58.635082 25481 leader_election.cc:290] T 37bde08107a44bac947149e44937c2e6 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:58.637928 22700 leader_election.cc:304] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:58.639200 22700 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:58.640377 22700 leader_election.cc:304] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:58.334550 25840 raft_consensus.cc:493] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:58.388765 25732 raft_consensus.cc:3055] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 10 LEADER]: Stepping down as leader of term 10
W20251026 14:59:58.397161 23369 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 130818us
I20251026 14:59:58.641728 25583 raft_consensus.cc:2749] T 3620632bd772435f97f799be96d493b8 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:58.336129 25758 raft_consensus.cc:493] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:58.642482 25583 raft_consensus.cc:2749] T ef8c3fc001e34801b3f872531b651e4d P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20251026 14:59:58.642410 25758 raft_consensus.cc:515] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.643222 25583 raft_consensus.cc:2749] T c9339fcac3bb4aa1949bdfb69eeedc2d P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20251026 14:59:58.430866 22771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:58.433634 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 1.96616e+06us
W20251026 14:59:58.644129 22624 leader_election.cc:336] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 2.233s (SENT)
W20251026 14:59:58.455816 22773 connection.cc:232] RPC call timeout handler was delayed by -1.8836012290000781s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1603s for a response
I20251026 14:59:58.437844 25241 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:58.644843 22773 connection.cc:232] RPC call timeout handler was delayed by -1.8701012290000563s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16180000000000003s for a response
W20251026 14:59:58.645227 22773 connection.cc:232] RPC call timeout handler was delayed by -1.8674012290000519s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16210000000000002s for a response
I20251026 14:59:58.645399 25830 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:58.467098 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ef8c3fc001e34801b3f872531b651e4d" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:58.646288 22655 raft_consensus.cc:2468] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 9.
W20251026 14:59:58.442968 25193 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.981s user 0.012s sys 0.051s
W20251026 14:59:58.646804 25193 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.185s user 0.013s sys 0.052s
I20251026 14:59:58.470202 25828 raft_consensus.cc:493] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:58.516264 25753 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:58.647331 25828 raft_consensus.cc:515] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:58.647441 25753 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:58.476409 22650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "988275d9b2574d0381d8f21ee8d99381" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
W20251026 14:59:58.488147 22772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:58.649780 25753 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185)
I20251026 14:59:58.522840 25604 raft_consensus.cc:493] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:58.524333 25824 raft_consensus.cc:493] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:58.650660 25604 raft_consensus.cc:515] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.650838 25824 raft_consensus.cc:515] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } }
I20251026 14:59:58.652838 25604 leader_election.cc:290] T 7de11d91eebd4cd0b3183e4c044d4004 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:58.527108 22653 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1719) took 2085 ms (client timeout 1553 ms). Trace:
I20251026 14:59:58.531832 25856 raft_consensus.cc:493] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
W20251026 14:59:58.533311 22625 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20251026 14:59:58.544530 25849 log.cc:927] Time spent T 129989aafdc84d6b8e56fe97c7182fff P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 1.991s user 0.007s sys 0.000s
I20251026 14:59:58.547719 22622 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:58.556902 25857 raft_consensus.cc:493] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:58.562542 22552 catalog_manager.cc:5649] T 9440e0456e0648a5af21108f5fd7cfba P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:58.565840 25586 raft_consensus.cc:493] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:58.566306 25860 raft_consensus.cc:493] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:58.599717 25862 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:58.599308 25861 raft_consensus.cc:493] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:58.620743 25851 log.cc:927] Time spent T 59b48d4137e3417983741d9cc5f92f42 P 0d684445287b450a8f738b6ad4b59821: Append to log took a long time: real 2.067s user 0.003s sys 0.004s
I20251026 14:59:58.625385 25865 raft_consensus.cc:493] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:58.607051 25709 consensus_queue.cc:1048] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
W20251026 14:59:58.636574 22587 debug-util.cc:398] Leaking SignalData structure 0x7b080004f5c0 after lost signal to thread 22758
I20251026 14:59:58.641538 25840 raft_consensus.cc:515] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.644434 25758 leader_election.cc:290] T c9edd43dec704b6e8351cebe3921d663 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:58.645048 25241 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.645927 25830 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:58.646589 22700 leader_election.cc:304] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:58.494618 22697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:58.496343 25527 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:58.503847 25529 consensus_meta.cc:287] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821: Time spent flushing consensus metadata: real 2.108s user 0.021s sys 0.002s
I20251026 14:59:58.537025 25855 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:58.538838 25850 log.cc:927] Time spent T 955c860c7f6f4f81bbe8e59e4c3dbcf5 P 562bb1e1c8e445b8baecd29c3669658e: Append to log took a long time: real 1.985s user 0.007s sys 0.000s
I20251026 14:59:58.678171 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1badbeef1a1c4427bbfa1b762c7d0739" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:58.676915 22804 raft_consensus.cc:1275] T 82201a13e5354c8ea3b41436df180125 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20251026 14:59:58.699119 22653 rpcz_store.cc:269] 1026 14:59:56.441378 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:56.441522 (+ 144us) service_pool.cc:225] Handling call
1026 14:59:58.526842 (+2085320us) spinlock_profiling.cc:243] Waited 61.8 ms on lock 0x7b540010d1d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:58.526996 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":142062464}
W20251026 14:59:58.700062 22587 debug-util.cc:398] Leaking SignalData structure 0x7b08003e7080 after lost signal to thread 22759
I20251026 14:59:58.678555 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3b18a8235bfa4d2cb35c7976ce46e14c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:58.679564 22803 raft_consensus.cc:1275] T 4e7ddd2458b64111839b7740daabf522 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Refusing update from remote peer df3a0d1619ee48e8ab68ee0543a72a29: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20251026 14:59:58.644521 22624 leader_election.cc:304] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
I20251026 14:59:58.649799 22650 raft_consensus.cc:2393] T 988275d9b2574d0381d8f21ee8d99381 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d684445287b450a8f738b6ad4b59821 in current term 10: Already voted for candidate df3a0d1619ee48e8ab68ee0543a72a29 in this term.
W20251026 14:59:58.636529 22658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54574 (request call id 1707) took 2357 ms (client timeout 2623 ms). Trace:
W20251026 14:59:58.645946 22773 connection.cc:232] RPC call timeout handler was delayed by -1.7819012290000273s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1716s for a response
W20251026 14:59:58.713656 22773 connection.cc:232] RPC call timeout handler was delayed by -1.7387012290000712s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1764s for a response
W20251026 14:59:58.714334 22587 debug-util.cc:398] Leaking SignalData structure 0x7b0800139760 after lost signal to thread 22832
W20251026 14:59:58.699608 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 94629.5us
W20251026 14:59:58.715072 22587 debug-util.cc:398] Leaking SignalData structure 0x7b08003b37e0 after lost signal to thread 22833
W20251026 14:59:58.715093 22773 connection.cc:232] RPC call timeout handler was delayed by -1.7342012290000639s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1769s for a response
W20251026 14:59:58.715373 22658 rpcz_store.cc:269] 1026 14:59:56.278720 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 14:59:56.278894 (+ 174us) service_pool.cc:225] Handling call
1026 14:59:58.636330 (+2357436us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b540010e0d4. stack: 00007fd8ce07fa23 00007fd8ce07f920 00007fd8cd9d2574 00007fd8d7679b14 00007fd8d7679aa9 00007fd8d10c74c1 00007fd8d1161e8a 00007fd8d61070d7 00007fd8cf6a4834 00007fd8cf6a47c2 00007fd8cf6a46f1 00007fd8cf6a466c 00007fd8cf6a3922 00007fd8cf49decc 00007fd8cf49d306 00007fd8cf49ccaf
1026 14:59:58.636472 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":318690176}
W20251026 14:59:58.717511 22587 debug-util.cc:398] Leaking SignalData structure 0x7b08000cea20 after lost signal to thread 23428
W20251026 14:59:58.717881 22587 debug-util.cc:398] Leaking SignalData structure 0x7b08002e5d00 after lost signal to thread 23429
W20251026 14:59:58.717922 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 170261us
W20251026 14:59:58.718446 22587 debug-util.cc:398] Leaking SignalData structure 0x7b08000ed7c0 after lost signal to thread 25621
W20251026 14:59:58.718911 22587 debug-util.cc:398] Leaking SignalData structure 0x7b0800336380 after lost signal to thread 25814
I20251026 14:59:58.703397 25857 raft_consensus.cc:515] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.716775 25527 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:58.719192 22587 debug-util.cc:398] Leaking SignalData structure 0x7b08002ab280 after lost signal to thread 25820
I20251026 14:59:58.720067 25855 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:58.719694 25828 leader_election.cc:290] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:58.702963 25861 raft_consensus.cc:515] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:58.653035 25824 leader_election.cc:290] T b0734967ed764a3c855f8eab30c49570 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:58.721766 25527 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:58.722117 25855 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:58.716917 25529 raft_consensus.cc:515] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.723073 25830 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:58.723196 25861 leader_election.cc:290] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:58.724570 25529 leader_election.cc:290] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:58.725163 25862 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:58.725723 25857 leader_election.cc:290] T c9339fcac3bb4aa1949bdfb69eeedc2d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:58.641767 25732 raft_consensus.cc:740] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:58.727409 25862 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:58.676774 25804 raft_consensus.cc:493] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:58.703224 25856 raft_consensus.cc:515] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.728297 25804 raft_consensus.cc:515] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.730257 25804 leader_election.cc:290] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:58.730849 22552 catalog_manager.cc:5649] T 37bde08107a44bac947149e44937c2e6 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 4 to 5, leader changed from df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 5 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:58.733783 25856 leader_election.cc:290] T 59b48d4137e3417983741d9cc5f92f42 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:58.734499 25241 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:58.716589 25586 raft_consensus.cc:515] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.702939 25860 raft_consensus.cc:515] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.736379 25586 leader_election.cc:290] T 143c213a935041868ad3b0d9584ad863 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:58.736899 25860 leader_election.cc:290] T 32b237245fe346a689e59ddc315b4577 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:58.727411 25732 consensus_queue.cc:260] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.738221 25840 leader_election.cc:290] T 82201a13e5354c8ea3b41436df180125 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:58.732580 22552 catalog_manager.cc:5649] T 3b18a8235bfa4d2cb35c7976ce46e14c P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 2 to 3, leader changed from 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2) to df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1). New cstate: current_term: 3 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:58.749109 22801 raft_consensus.cc:2393] T 1badbeef1a1c4427bbfa1b762c7d0739 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 in current term 9: Already voted for candidate 0d684445287b450a8f738b6ad4b59821 in this term.
I20251026 14:59:58.653460 25604 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:58.654049 25193 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:58.758205 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:58.742326 22774 leader_election.cc:304] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:58.752508 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:58.753139 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:58.750434 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:58.770171 22804 raft_consensus.cc:2468] T 02799da19b36403499a63101b8418ffe P 0d684445287b450a8f738b6ad4b59821 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 2.
I20251026 14:59:58.673779 25196 raft_consensus.cc:493] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:58.760299 22552 catalog_manager.cc:5649] T fbb846c2db3b4f44b2245a7493b36c7c P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 1 to 7. New cstate: current_term: 7 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
W20251026 14:59:58.766297 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 122195us
I20251026 14:59:58.757833 25758 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:58.760488 25840 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:58.763461 25732 raft_consensus.cc:3060] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Advancing to term 11
I20251026 14:59:58.753597 25861 raft_consensus.cc:493] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:58.758950 25481 raft_consensus.cc:2749] T 9423e278399742f48ca9ee203d7f92cb P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20251026 14:59:58.757620 25857 raft_consensus.cc:3060] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:58.770350 22803 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 7.
I20251026 14:59:58.757161 25824 raft_consensus.cc:2804] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20251026 14:59:58.771351 22624 leader_election.cc:336] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 2.246s (SENT)
I20251026 14:59:58.777801 22624 leader_election.cc:304] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:58.778373 22698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251026 14:59:58.778482 25709 raft_consensus.cc:2804] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20251026 14:59:58.778920 25709 raft_consensus.cc:493] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:58.779276 25709 raft_consensus.cc:3060] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Advancing to term 10
I20251026 14:59:58.739837 25865 raft_consensus.cc:515] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.758021 25604 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.783560 25865 leader_election.cc:290] T 3103efcd5601429a95eeb7a270f6fab1 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:58.784209 25604 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:58.789430 25824 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:58.790238 25824 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Advancing to term 10
W20251026 14:59:58.771595 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 52928.1us
I20251026 14:59:58.773458 23367 leader_election.cc:304] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters:
I20251026 14:59:58.793541 25862 raft_consensus.cc:493] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:58.794550 23367 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters:
I20251026 14:59:58.794350 25862 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.795501 25855 raft_consensus.cc:2804] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20251026 14:59:58.795955 25855 raft_consensus.cc:493] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 562bb1e1c8e445b8baecd29c3669658e)
I20251026 14:59:58.796343 25855 raft_consensus.cc:3060] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 3 FOLLOWER]: Advancing to term 4
I20251026 14:59:58.776235 25861 raft_consensus.cc:515] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.775422 25758 raft_consensus.cc:515] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.777926 25840 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:58.774013 22552 catalog_manager.cc:5649] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
W20251026 14:59:58.774811 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 128319us
W20251026 14:59:58.776757 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 59203.4us
I20251026 14:59:58.764034 25193 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:58.772580 25196 raft_consensus.cc:515] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.797209 25862 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:58.800967 25830 raft_consensus.cc:2804] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20251026 14:59:58.821563 25709 raft_consensus.cc:515] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.822395 25830 raft_consensus.cc:493] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:58.822793 25830 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:58.823863 25709 leader_election.cc:290] T 2979554f0ac542ab8cfcb32c7d2778d5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:58.799216 25861 leader_election.cc:290] T 6b172938cfc843318b60a7fc78190d2c P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:58.804317 25758 leader_election.cc:290] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:58.814205 22700 leader_election.cc:304] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:58.818552 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:58.830276 23402 raft_consensus.cc:2468] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 6.
I20251026 14:59:58.829471 25861 raft_consensus.cc:493] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:58.830741 22700 leader_election.cc:304] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:58.831661 22771 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters:
I20251026 14:59:58.831965 22700 leader_election.cc:304] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
W20251026 14:59:58.832476 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 84387.7us
I20251026 14:59:58.832842 25481 raft_consensus.cc:2749] T dea6ba9f8ecc4124bfedb6febafde72b P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251026 14:59:58.832600 25830 raft_consensus.cc:515] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:58.833007 22773 connection.cc:232] RPC call timeout handler was delayed by -1.6883012289999897s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18200000000000002s for a response
W20251026 14:59:58.817770 22624 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.2:46507, user_credentials={real_user=slave}} blocked reactor thread for 46453.3us
I20251026 14:59:58.818423 25869 raft_consensus.cc:2749] T 6a1f6f52f31d4a9e98b2c86f2358ea8b P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:58.818616 25193 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:58.819065 22552 catalog_manager.cc:5649] T ea1e7da57b65433b9e01cad8c4adeb75 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 2 to 3, leader changed from 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4) to 0d684445287b450a8f738b6ad4b59821 (127.21.2.3). New cstate: current_term: 3 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 14:59:58.836156 25783 raft_consensus.cc:2749] T 6d267e50950c4ef2b95c745e7103cafb P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:58.816090 22802 raft_consensus.cc:2410] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 248493ec312e4398b828c6cb00e5c4b1 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20251026 14:59:58.845780 25857 raft_consensus.cc:2749] T 37bde08107a44bac947149e44937c2e6 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20251026 14:59:58.847580 25709 raft_consensus.cc:2804] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20251026 14:59:58.837150 22700 leader_election.cc:304] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:58.848560 25830 leader_election.cc:290] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 8 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:58.849128 22700 leader_election.cc:304] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:58.848138 22552 catalog_manager.cc:5649] T 3ea63385804c40689502aefb2b695e78 P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "0d684445287b450a8f738b6ad4b59821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } health_report { overall_health: HEALTHY } } }
I20251026 14:59:58.850611 25241 raft_consensus.cc:2804] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:58.850409 22552 catalog_manager.cc:5649] T c9339fcac3bb4aa1949bdfb69eeedc2d P 0d684445287b450a8f738b6ad4b59821 reported cstate change: term changed from 4 to 7. New cstate: current_term: 7 leader_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } }
I20251026 14:59:58.836190 25481 raft_consensus.cc:2749] T 4dea3b45142b41baacfa105498c9e04e P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251026 14:59:58.851208 25241 raft_consensus.cc:697] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [term 7 LEADER]: Becoming Leader. State: Replica: 562bb1e1c8e445b8baecd29c3669658e, State: Running, Role: LEADER
I20251026 14:59:58.851405 25783 raft_consensus.cc:2749] T 8bb7c83ff5ae4d4a802850a40803bc55 P 562bb1e1c8e445b8baecd29c3669658e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20251026 14:59:58.829106 22697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 72606.4us
I20251026 14:59:58.852172 25241 consensus_queue.cc:237] T 9ab2875267d54ec89df26641d40fbb40 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.855485 22732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e"
I20251026 14:59:58.855947 22728 raft_consensus.cc:1240] T ea1e7da57b65433b9e01cad8c4adeb75 P 562bb1e1c8e445b8baecd29c3669658e [term 6 FOLLOWER]: Rejecting Update request from peer 248493ec312e4398b828c6cb00e5c4b1 for earlier term 5. Current term is 6. Ops: []
I20251026 14:59:58.856046 22732 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:58.856866 25824 raft_consensus.cc:515] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.820809 25196 leader_election.cc:290] T 5618ccace02e47938463dce9ca26e300 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:58.860132 23402 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a673ec034b704e8e86556afe9322d8c2" candidate_uuid: "562bb1e1c8e445b8baecd29c3669658e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:58.860172 22802 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821"
I20251026 14:59:58.861198 22802 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:58.848125 25709 raft_consensus.cc:493] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:58.862053 25709 raft_consensus.cc:3060] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:58.819883 22774 leader_election.cc:304] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:58.818837 22622 leader_election.cc:304] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:58.863109 22622 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.3:39645, user_credentials={real_user=slave}} blocked reactor thread for 44353.7us
I20251026 14:59:58.807441 25840 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
W20251026 14:59:58.842285 22773 connection.cc:232] RPC call timeout handler was delayed by -1.6266804120000415s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2097s for a response
I20251026 14:59:58.865495 25862 raft_consensus.cc:2804] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 FOLLOWER]: Leader election won for term 9
W20251026 14:59:58.865836 22773 connection.cc:232] RPC call timeout handler was delayed by -1.6086628619999601s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3002s for a response
I20251026 14:59:58.866081 25862 raft_consensus.cc:697] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [term 9 LEADER]: Becoming Leader. State: Replica: df3a0d1619ee48e8ab68ee0543a72a29, State: Running, Role: LEADER
W20251026 14:59:58.866523 22773 connection.cc:232] RPC call timeout handler was delayed by -1.5639902359999951s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1825s for a response
W20251026 14:59:58.866926 22773 connection.cc:232] RPC call timeout handler was delayed by -1.492169874999945s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32120000000000004s for a response
W20251026 14:59:58.867236 22773 connection.cc:232] RPC call timeout handler was delayed by -1.4624012290000792s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20710000000000003s for a response
W20251026 14:59:58.867944 22773 connection.cc:232] RPC call timeout handler was delayed by -1.43810122900004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2098s for a response
W20251026 14:59:58.868389 22773 connection.cc:232] RPC call timeout handler was delayed by -1.3461264600000504s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16210000000000002s for a response
I20251026 14:59:58.868490 25702 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:58.868623 25196 raft_consensus.cc:493] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251026 14:59:58.868788 22773 connection.cc:232] RPC call timeout handler was delayed by -1.3445012290000022s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2202s for a response
W20251026 14:59:58.842866 22624 leader_election.cc:336] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 2.249s (SENT)
I20251026 14:59:58.867154 25862 consensus_queue.cc:237] T c9edd43dec704b6e8351cebe3921d663 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.869475 22624 leader_election.cc:304] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821, 562bb1e1c8e445b8baecd29c3669658e
W20251026 14:59:58.869246 22773 connection.cc:232] RPC call timeout handler was delayed by -1.3310012289999804s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2217s for a response
W20251026 14:59:58.871227 22773 connection.cc:232] RPC call timeout handler was delayed by -1.2741792479999958s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20120000000000002s for a response
W20251026 14:59:58.871516 22773 connection.cc:232] RPC call timeout handler was delayed by -1.2653550160000577s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1602s for a response
W20251026 14:59:58.871800 22773 connection.cc:232] RPC call timeout handler was delayed by -1.224503575999961s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1553s for a response
W20251026 14:59:58.872051 22773 connection.cc:232] RPC call timeout handler was delayed by -1.2075264600000537s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1775s for a response
W20251026 14:59:58.872296 22773 connection.cc:232] RPC call timeout handler was delayed by -1.1706902360000413s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2262s for a response
I20251026 14:59:58.872539 25871 consensus_queue.cc:1048] T 4e7ddd2458b64111839b7740daabf522 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20251026 14:59:58.874164 25865 consensus_queue.cc:1048] T 82201a13e5354c8ea3b41436df180125 P df3a0d1619ee48e8ab68ee0543a72a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 14:59:58.874475 22653 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2efb26bccebf43b5b8a85f51fd977714" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:58.875133 22653 raft_consensus.cc:3060] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:58.875764 25865 raft_consensus.cc:2749] T 2e6d661a639541b3a40bf3c1fc6c0a88 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:58.876483 25865 raft_consensus.cc:493] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:58.877063 25824 leader_election.cc:290] T ef8c3fc001e34801b3f872531b651e4d P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 10 election: Requested vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:58.876956 25865 raft_consensus.cc:515] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:58.879863 22650 raft_consensus.cc:1275] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 10 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20251026 14:59:58.880380 25865 leader_election.cc:290] T a673ec034b704e8e86556afe9322d8c2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:58.881001 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:58.881733 22655 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 7.
I20251026 14:59:58.869218 25196 raft_consensus.cc:515] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.884613 25196 leader_election.cc:290] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:58.885804 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:58.886606 22656 raft_consensus.cc:2468] T 5a0e9c0787aa4053afa2791a86bb263a P df3a0d1619ee48e8ab68ee0543a72a29 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 5.
I20251026 14:59:58.889767 22802 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 8.
I20251026 14:59:58.837759 25732 consensus_queue.cc:1048] T 5618ccace02e47938463dce9ca26e300 P 562bb1e1c8e445b8baecd29c3669658e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 14:59:58.898561 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32b237245fe346a689e59ddc315b4577" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:58.898862 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3103efcd5601429a95eeb7a270f6fab1" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:58.899228 22807 raft_consensus.cc:2468] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
I20251026 14:59:58.859165 22727 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02799da19b36403499a63101b8418ffe" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "562bb1e1c8e445b8baecd29c3669658e" is_pre_election: true
I20251026 14:59:58.860656 22731 raft_consensus.cc:1275] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: Refusing update from remote peer 248493ec312e4398b828c6cb00e5c4b1: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20251026 14:59:58.900591 22731 pending_rounds.cc:85] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: Aborting all ops after (but not including) 8
I20251026 14:59:58.900849 22731 pending_rounds.cc:107] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20251026 14:59:58.860301 25830 consensus_queue.cc:1059] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20251026 14:59:58.901932 25830 raft_consensus.cc:3055] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 14:59:58.902273 25830 raft_consensus.cc:740] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:58.903083 25830 consensus_queue.cc:260] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:58.904431 25830 raft_consensus.cc:3060] T ea1e7da57b65433b9e01cad8c4adeb75 P 248493ec312e4398b828c6cb00e5c4b1 [term 5 FOLLOWER]: Advancing to term 6
I20251026 14:59:58.869060 25702 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.906831 25702 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507)
I20251026 14:59:58.891577 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:58.892277 22803 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0734967ed764a3c855f8eab30c49570" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:58.892518 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:58.908461 22803 raft_consensus.cc:2410] T b0734967ed764a3c855f8eab30c49570 P 0d684445287b450a8f738b6ad4b59821 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20251026 14:59:58.908929 22799 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6b172938cfc843318b60a7fc78190d2c" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:58.901077 22731 raft_consensus.cc:2889] T a673ec034b704e8e86556afe9322d8c2 P 562bb1e1c8e445b8baecd29c3669658e [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 14:59:58.909614 22799 raft_consensus.cc:2468] T 6b172938cfc843318b60a7fc78190d2c P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
I20251026 14:59:58.909790 25702 raft_consensus.cc:2804] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader election won for term 9
I20251026 14:59:58.892971 22804 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7de11d91eebd4cd0b3183e4c044d4004" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:58.910382 25702 raft_consensus.cc:697] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [term 9 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:58.910550 22804 raft_consensus.cc:2468] T 7de11d91eebd4cd0b3183e4c044d4004 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 8.
I20251026 14:59:58.893157 22801 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9339fcac3bb4aa1949bdfb69eeedc2d" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:58.911437 25709 raft_consensus.cc:515] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:58.831045 25861 raft_consensus.cc:515] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.896433 22805 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59b48d4137e3417983741d9cc5f92f42" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:58.898162 22806 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "143c213a935041868ad3b0d9584ad863" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:58.900063 22807 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ab2875267d54ec89df26641d40fbb40" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:58.913233 22807 raft_consensus.cc:2468] T 9ab2875267d54ec89df26641d40fbb40 P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 7.
W20251026 14:59:58.909627 22774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 89821.1us
I20251026 14:59:58.914489 22774 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821; no voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:58.915771 23403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "0d684445287b450a8f738b6ad4b59821" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1"
I20251026 14:59:58.916366 23403 raft_consensus.cc:3060] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 6 FOLLOWER]: Advancing to term 7
I20251026 14:59:58.909941 22656 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "741e08995c924c8e9ea37c02c41a5bc7" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29"
I20251026 14:59:58.916572 25193 consensus_queue.cc:1048] T e2b59eefda3b4f2d93934f02c4981c62 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:59:58.917248 22656 raft_consensus.cc:3060] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:58.894488 22808 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:58.918879 22808 raft_consensus.cc:2368] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df3a0d1619ee48e8ab68ee0543a72a29 for earlier term 6. Current term is 7.
I20251026 14:59:58.919718 25193 consensus_queue.cc:1048] T f688f64ce4e84d419d88768435bf773f P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 14:59:58.911388 25702 consensus_queue.cc:237] T 32b237245fe346a689e59ddc315b4577 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.895311 23400 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a0e9c0787aa4053afa2791a86bb263a" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "248493ec312e4398b828c6cb00e5c4b1" is_pre_election: true
I20251026 14:59:58.900115 22800 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9339edd1ecbf42e3a5e953b2b47f52f2" candidate_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0d684445287b450a8f738b6ad4b59821" is_pre_election: true
I20251026 14:59:58.922119 22800 raft_consensus.cc:2468] T 9339edd1ecbf42e3a5e953b2b47f52f2 P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df3a0d1619ee48e8ab68ee0543a72a29 in term 9.
I20251026 14:59:58.919664 25534 consensus_queue.cc:1048] T a35eb71706aa4f97bf6c64430916aef5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20251026 14:59:58.850706 22700 leader_election.cc:304] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
I20251026 14:59:58.909649 22655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a35eb71706aa4f97bf6c64430916aef5" candidate_uuid: "248493ec312e4398b828c6cb00e5c4b1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" is_pre_election: true
I20251026 14:59:58.923521 22655 raft_consensus.cc:2468] T a35eb71706aa4f97bf6c64430916aef5 P df3a0d1619ee48e8ab68ee0543a72a29 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 7.
I20251026 14:59:58.839615 25855 raft_consensus.cc:515] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
W20251026 14:59:58.872545 22773 connection.cc:232] RPC call timeout handler was delayed by -1.1609550160000026s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1718s for a response
I20251026 14:59:58.925246 23367 leader_election.cc:304] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters:
W20251026 14:59:58.929651 22773 connection.cc:232] RPC call timeout handler was delayed by -1.0691550159999679s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18200000000000002s for a response
W20251026 14:59:58.930070 22773 connection.cc:232] RPC call timeout handler was delayed by -1.0528160049999542s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1507s for a response
I20251026 14:59:58.906632 22732 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P 562bb1e1c8e445b8baecd29c3669658e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 8.
I20251026 14:59:58.932572 23369 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1, 562bb1e1c8e445b8baecd29c3669658e; no voters:
W20251026 14:59:58.931949 22624 connection.cc:232] RPC call timeout handler was delayed by -2.43135307s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1519s for a response
I20251026 14:59:58.934639 23403 raft_consensus.cc:2468] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d684445287b450a8f738b6ad4b59821 in term 7.
I20251026 14:59:58.934854 22653 raft_consensus.cc:2468] T 2efb26bccebf43b5b8a85f51fd977714 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 8.
I20251026 14:59:58.937577 22771 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1; no voters:
I20251026 14:59:58.938966 22774 leader_election.cc:304] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:58.939064 22624 connection.cc:232] RPC call timeout handler was delayed by -2.2684530700000778s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17s for a response
W20251026 14:59:58.939770 22624 connection.cc:232] RPC call timeout handler was delayed by -2.0194895890000453s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1529s for a response
W20251026 14:59:58.940181 22624 connection.cc:232] RPC call timeout handler was delayed by -1.9555705370000851s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1836s for a response
I20251026 14:59:58.946588 25857 raft_consensus.cc:493] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 14:59:58.947163 25857 raft_consensus.cc:515] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.949261 25857 leader_election.cc:290] T dea6ba9f8ecc4124bfedb6febafde72b P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:58.923144 25527 consensus_queue.cc:1048] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.026s
I20251026 14:59:58.950573 25527 consensus_queue.cc:1048] T a673ec034b704e8e86556afe9322d8c2 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
W20251026 14:59:58.945417 22624 connection.cc:232] RPC call timeout handler was delayed by -1.9213705370000298s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1874s for a response
W20251026 14:59:58.952598 22624 connection.cc:232] RPC call timeout handler was delayed by -1.8277895889999627s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17420000000000002s for a response
I20251026 14:59:58.952666 25866 raft_consensus.cc:2804] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20251026 14:59:58.953017 22624 connection.cc:232] RPC call timeout handler was delayed by -1.8052705370000695s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20030000000000003s for a response
I20251026 14:59:58.953147 25866 raft_consensus.cc:493] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251026 14:59:58.953380 22624 connection.cc:232] RPC call timeout handler was delayed by -1.6953673990001334s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1526s for a response
I20251026 14:59:58.953555 25866 raft_consensus.cc:3060] T ef8c3fc001e34801b3f872531b651e4d P 0d684445287b450a8f738b6ad4b59821 [term 9 FOLLOWER]: Advancing to term 10
W20251026 14:59:58.954527 22624 connection.cc:232] RPC call timeout handler was delayed by -1.5378673990001062s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17010000000000003s for a response
W20251026 14:59:58.954605 22700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 103972us
W20251026 14:59:58.955233 22624 connection.cc:232] RPC call timeout handler was delayed by -1.4865673990001369s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1758s for a response
W20251026 14:59:58.955879 22624 connection.cc:232] RPC call timeout handler was delayed by -1.4640673990001005s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17830000000000001s for a response
W20251026 14:59:58.956163 22624 connection.cc:232] RPC call timeout handler was delayed by -1.3695673990000614s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1888s for a response
W20251026 14:59:58.956400 22624 connection.cc:232] RPC call timeout handler was delayed by -1.3578673990000425s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19010000000000002s for a response
I20251026 14:59:58.958096 22774 leader_election.cc:304] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
W20251026 14:59:58.958611 22624 connection.cc:232] RPC call timeout handler was delayed by -1.3167257909999535s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15980000000000003s for a response
W20251026 14:59:58.958968 22624 connection.cc:232] RPC call timeout handler was delayed by -1.259125790999974s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16620000000000001s for a response
I20251026 14:59:58.930807 25709 leader_election.cc:290] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 7 election: Requested vote from peers df3a0d1619ee48e8ab68ee0543a72a29 (127.21.2.1:46185), 248493ec312e4398b828c6cb00e5c4b1 (127.21.2.4:43341)
I20251026 14:59:58.930425 25196 raft_consensus.cc:2749] T 741e08995c924c8e9ea37c02c41a5bc7 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251026 14:59:58.957521 25604 raft_consensus.cc:493] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:58.965957 25604 raft_consensus.cc:515] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.968238 25604 leader_election.cc:290] T 8bb7c83ff5ae4d4a802850a40803bc55 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:58.957348 25193 raft_consensus.cc:2804] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 7 FOLLOWER]: Leader election won for term 7
I20251026 14:59:58.962098 25857 raft_consensus.cc:493] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d684445287b450a8f738b6ad4b59821)
I20251026 14:59:58.969780 25857 raft_consensus.cc:515] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.971693 25857 leader_election.cc:290] T ba8784cbec044fb7b5b098961f0f94b2 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
I20251026 14:59:58.972268 25534 raft_consensus.cc:2804] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 11 FOLLOWER]: Leader election won for term 11
I20251026 14:59:58.972330 25193 raft_consensus.cc:697] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [term 7 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:58.972874 25534 raft_consensus.cc:697] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [term 11 LEADER]: Becoming Leader. State: Replica: 0d684445287b450a8f738b6ad4b59821, State: Running, Role: LEADER
I20251026 14:59:58.973460 25193 consensus_queue.cc:237] T 5a0e9c0787aa4053afa2791a86bb263a P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:58.973762 25534 consensus_queue.cc:237] T 37e63cca2eb94ba8ac9b6c8cdf99c4a5 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "562bb1e1c8e445b8baecd29c3669658e" member_type: VOTER last_known_addr { host: "127.21.2.2" port: 46507 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } }
I20251026 14:59:58.944952 25732 raft_consensus.cc:2804] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election won for term 8
W20251026 14:59:58.959506 22624 connection.cc:232] RPC call timeout handler was delayed by -1.1949673990001202s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2082s for a response
W20251026 14:59:58.959770 22773 leader_election.cc:336] T 143c213a935041868ad3b0d9584ad863 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507): Timed out: RequestConsensusVote RPC to 127.21.2.2:46507 timed out after 1.543s (SENT)
I20251026 14:59:58.960357 22700 leader_election.cc:304] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:58.962364 23402 raft_consensus.cc:1240] T 02799da19b36403499a63101b8418ffe P 248493ec312e4398b828c6cb00e5c4b1 [term 4 FOLLOWER]: Rejecting Update request from peer 562bb1e1c8e445b8baecd29c3669658e for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20251026 14:59:58.939245 25804 consensus_queue.cc:1048] T e923fc1e1fa340a7b482357487981b39 P 0d684445287b450a8f738b6ad4b59821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20251026 14:59:58.938776 25828 raft_consensus.cc:2804] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 8 FOLLOWER]: Leader election won for term 8
I20251026 14:59:58.977231 22774 leader_election.cc:304] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0d684445287b450a8f738b6ad4b59821, 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters:
I20251026 14:59:58.980019 22656 raft_consensus.cc:2468] T 741e08995c924c8e9ea37c02c41a5bc7 P df3a0d1619ee48e8ab68ee0543a72a29 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248493ec312e4398b828c6cb00e5c4b1 in term 8.
I20251026 14:59:58.981506 25529 raft_consensus.cc:2764] T 2efb26bccebf43b5b8a85f51fd977714 P 0d684445287b450a8f738b6ad4b59821 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20251026 14:59:58.976413 25732 raft_consensus.cc:493] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 14:59:58.990861 25732 raft_consensus.cc:3060] T e2b59eefda3b4f2d93934f02c4981c62 P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Advancing to term 8
I20251026 14:59:58.991523 22700 leader_election.cc:304] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562bb1e1c8e445b8baecd29c3669658e; no voters: 0d684445287b450a8f738b6ad4b59821, df3a0d1619ee48e8ab68ee0543a72a29
I20251026 14:59:58.992092 25828 raft_consensus.cc:697] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [term 8 LEADER]: Becoming Leader. State: Replica: 248493ec312e4398b828c6cb00e5c4b1, State: Running, Role: LEADER
I20251026 14:59:58.992936 25753 raft_consensus.cc:2749] T 4129daa4fb70408193f07b8e6d25effe P 562bb1e1c8e445b8baecd29c3669658e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20251026 14:59:58.993129 25828 consensus_queue.cc:237] T 741e08995c924c8e9ea37c02c41a5bc7 P 248493ec312e4398b828c6cb00e5c4b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "df3a0d1619ee48e8ab68ee0543a72a29" member_type: VOTER last_known_addr { host: "127.21.2.1" port: 46185 } } peers { permanent_uuid: "0d684445287b450a8f738b6ad4b59821" member_type: VOTER last_known_addr { host: "127.21.2.3" port: 39645 } } peers { permanent_uuid: "248493ec312e4398b828c6cb00e5c4b1" member_type: VOTER last_known_addr { host: "127.21.2.4" port: 43341 } attrs { promote: false } }
I20251026 14:59:58.993954 25753 raft_consensus.cc:2749] T fbb846c2db3b4f44b2245a7493b36c7c P 562bb1e1c8e445b8baecd29c3669658e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20251026 14:59:58.954843 25858 raft_consensus.cc:2804] T 2efb26bccebf43b5b8a85f51fd977714 P 248493ec312e4398b828c6cb00e5c4b1 [term 8 FOLLOWER]: Leader election won for term 8
W20251026 14:59:58.995285 23370 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.2.1:46185, user_credentials={real_user=slave}} blocked reactor thread for 58755.9us
I20251026 14:59:58.996431 23370 leader_election.cc:304] T 5a0e9c0787aa4053afa2791a86bb263a P 248493ec312e4398b828c6cb00e5c4b1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 248493ec312e4398b828c6cb00e5c4b1, df3a0d1619ee48e8ab68ee0543a72a29; no voters: 0d684445287b450a8f738b6ad4b59821
W20251026 14:59:58.977315 22624 connection.cc:232] RPC call timeout handler was delayed by -1.1769673990000911s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2102s for a response
W20251026 14:59:58.998531 22624 connection.cc:232] RPC call timeout handler was delayed by -1.0188257910000402s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19290000000000002s for a response
I20251026 14:59:58.951620 25861 leader_election.cc:290] T e923fc1e1fa340a7b482357487981b39 P df3a0d1619ee48e8ab68ee0543a72a29 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 562bb1e1c8e445b8baecd29c3669658e (127.21.2.2:46507), 0d684445287b450a8f738b6ad4b59821 (127.21.2.3:39645)
W20251026 14:59:58.998955 22624 connection.cc:232] RPC call timeout handler was delayed by -1.0051705370000263s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2892s for a response
W20251026 14:59:59.006381 22773 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote ->